diff options
author | cinap_lenrek <cinap_lenrek@localhost> | 2011-05-03 11:25:13 +0000 |
---|---|---|
committer | cinap_lenrek <cinap_lenrek@localhost> | 2011-05-03 11:25:13 +0000 |
commit | 458120dd40db6b4df55a4e96b650e16798ef06a0 (patch) | |
tree | 8f82685be24fef97e715c6f5ca4c68d34d5074ee /sys/src/cmd/python/Doc/lib/liburllib.tex | |
parent | 3a742c699f6806c1145aea5149bf15de15a0afd7 (diff) |
add hg and python
Diffstat (limited to 'sys/src/cmd/python/Doc/lib/liburllib.tex')
-rw-r--r-- | sys/src/cmd/python/Doc/lib/liburllib.tex | 497 |
1 files changed, 497 insertions, 0 deletions
diff --git a/sys/src/cmd/python/Doc/lib/liburllib.tex b/sys/src/cmd/python/Doc/lib/liburllib.tex new file mode 100644 index 000000000..75ee310d5 --- /dev/null +++ b/sys/src/cmd/python/Doc/lib/liburllib.tex @@ -0,0 +1,497 @@ +\section{\module{urllib} --- + Open arbitrary resources by URL} + +\declaremodule{standard}{urllib} +\modulesynopsis{Open an arbitrary network resource by URL (requires sockets).} + +\index{WWW} +\index{World Wide Web} +\index{URL} + + +This module provides a high-level interface for fetching data across +the World Wide Web. In particular, the \function{urlopen()} function +is similar to the built-in function \function{open()}, but accepts +Universal Resource Locators (URLs) instead of filenames. Some +restrictions apply --- it can only open URLs for reading, and no seek +operations are available. + +It defines the following public functions: + +\begin{funcdesc}{urlopen}{url\optional{, data\optional{, proxies}}} +Open a network object denoted by a URL for reading. If the URL does +not have a scheme identifier, or if it has \file{file:} as its scheme +identifier, this opens a local file (without universal newlines); +otherwise it opens a socket to a server somewhere on the network. If +the connection cannot be made +the \exception{IOError} exception is raised. If all went well, a +file-like object is returned. This supports the following methods: +\method{read()}, \method{readline()}, \method{readlines()}, \method{fileno()}, +\method{close()}, \method{info()} and \method{geturl()}. It also has +proper support for the iterator protocol. +One caveat: the \method{read()} method, if the size argument is +omitted or negative, may not read until the end of the data stream; +there is no good way to determine that the entire stream from a socket +has been read in the general case. + +Except for the \method{info()} and \method{geturl()} methods, +these methods have the same interface as for +file objects --- see section \ref{bltin-file-objects} in this +manual. (It is not a built-in file object, however, so it can't be +used at those few places where a true built-in file object is +required.) + +The \method{info()} method returns an instance of the class +\class{mimetools.Message} containing meta-information associated +with the URL. When the method is HTTP, these headers are those +returned by the server at the head of the retrieved HTML page +(including Content-Length and Content-Type). When the method is FTP, +a Content-Length header will be present if (as is now usual) the +server passed back a file length in response to the FTP retrieval +request. A Content-Type header will be present if the MIME type can +be guessed. When the method is local-file, returned headers will include +a Date representing the file's last-modified time, a Content-Length +giving file size, and a Content-Type containing a guess at the file's +type. See also the description of the +\refmodule{mimetools}\refstmodindex{mimetools} module. + +The \method{geturl()} method returns the real URL of the page. In +some cases, the HTTP server redirects a client to another URL. The +\function{urlopen()} function handles this transparently, but in some +cases the caller needs to know which URL the client was redirected +to. The \method{geturl()} method can be used to get at this +redirected URL. + +If the \var{url} uses the \file{http:} scheme identifier, the optional +\var{data} argument may be given to specify a \code{POST} request +(normally the request type is \code{GET}). The \var{data} argument +must be in standard \mimetype{application/x-www-form-urlencoded} format; +see the \function{urlencode()} function below. + +The \function{urlopen()} function works transparently with proxies +which do not require authentication. In a \UNIX{} or Windows +environment, set the \envvar{http_proxy}, \envvar{ftp_proxy} or +\envvar{gopher_proxy} environment variables to a URL that identifies +the proxy server before starting the Python interpreter. For example +(the \character{\%} is the command prompt): + +\begin{verbatim} +% http_proxy="http://www.someproxy.com:3128" +% export http_proxy +% python +... +\end{verbatim} + +In a Windows environment, if no proxy environment variables are set, +proxy settings are obtained from the registry's Internet Settings +section. + +In a Macintosh environment, \function{urlopen()} will retrieve proxy +information from Internet\index{Internet Config} Config. + +Alternatively, the optional \var{proxies} argument may be used to +explicitly specify proxies. It must be a dictionary mapping scheme +names to proxy URLs, where an empty dictionary causes no proxies to be +used, and \code{None} (the default value) causes environmental proxy +settings to be used as discussed above. For example: + +\begin{verbatim} +# Use http://www.someproxy.com:3128 for http proxying +proxies = {'http': 'http://www.someproxy.com:3128'} +filehandle = urllib.urlopen(some_url, proxies=proxies) +# Don't use any proxies +filehandle = urllib.urlopen(some_url, proxies={}) +# Use proxies from environment - both versions are equivalent +filehandle = urllib.urlopen(some_url, proxies=None) +filehandle = urllib.urlopen(some_url) +\end{verbatim} + +The \function{urlopen()} function does not support explicit proxy +specification. If you need to override environmental proxy settings, +use \class{URLopener}, or a subclass such as \class{FancyURLopener}. + +Proxies which require authentication for use are not currently +supported; this is considered an implementation limitation. + +\versionchanged[Added the \var{proxies} support]{2.3} +\end{funcdesc} + +\begin{funcdesc}{urlretrieve}{url\optional{, filename\optional{, + reporthook\optional{, data}}}} +Copy a network object denoted by a URL to a local file, if necessary. +If the URL points to a local file, or a valid cached copy of the +object exists, the object is not copied. Return a tuple +\code{(\var{filename}, \var{headers})} where \var{filename} is the +local file name under which the object can be found, and \var{headers} +is whatever the \method{info()} method of the object returned by +\function{urlopen()} returned (for a remote object, possibly cached). +Exceptions are the same as for \function{urlopen()}. + +The second argument, if present, specifies the file location to copy +to (if absent, the location will be a tempfile with a generated name). +The third argument, if present, is a hook function that will be called +once on establishment of the network connection and once after each +block read thereafter. The hook will be passed three arguments; a +count of blocks transferred so far, a block size in bytes, and the +total size of the file. The third argument may be \code{-1} on older +FTP servers which do not return a file size in response to a retrieval +request. + +If the \var{url} uses the \file{http:} scheme identifier, the optional +\var{data} argument may be given to specify a \code{POST} request +(normally the request type is \code{GET}). The \var{data} argument +must in standard \mimetype{application/x-www-form-urlencoded} format; +see the \function{urlencode()} function below. + +\versionchanged[ +\function{urlretrieve()} will raise \exception{ContentTooShortError} +when it detects that the amount of data available +was less than the expected amount (which is the size reported by a +\var{Content-Length} header). This can occur, for example, when the +download is interrupted. + +The \var{Content-Length} is treated as a lower bound: if there's more data +to read, urlretrieve reads more data, but if less data is available, +it raises the exception. + +You can still retrieve the downloaded data in this case, it is stored +in the \member{content} attribute of the exception instance. + +If no \var{Content-Length} header was supplied, urlretrieve can +not check the size of the data it has downloaded, and just returns it. +In this case you just have to assume that the download was successful]{2.5} + +\end{funcdesc} + +\begin{datadesc}{_urlopener} +The public functions \function{urlopen()} and +\function{urlretrieve()} create an instance of the +\class{FancyURLopener} class and use it to perform their requested +actions. To override this functionality, programmers can create a +subclass of \class{URLopener} or \class{FancyURLopener}, then assign +an instance of that class to the +\code{urllib._urlopener} variable before calling the desired function. +For example, applications may want to specify a different +\mailheader{User-Agent} header than \class{URLopener} defines. This +can be accomplished with the following code: + +\begin{verbatim} +import urllib + +class AppURLopener(urllib.FancyURLopener): + version = "App/1.7" + +urllib._urlopener = AppURLopener() +\end{verbatim} +\end{datadesc} + +\begin{funcdesc}{urlcleanup}{} +Clear the cache that may have been built up by previous calls to +\function{urlretrieve()}. +\end{funcdesc} + +\begin{funcdesc}{quote}{string\optional{, safe}} +Replace special characters in \var{string} using the \samp{\%xx} escape. +Letters, digits, and the characters \character{_.-} are never quoted. +The optional \var{safe} parameter specifies additional characters +that should not be quoted --- its default value is \code{'/'}. + +Example: \code{quote('/\~{}connolly/')} yields \code{'/\%7econnolly/'}. +\end{funcdesc} + +\begin{funcdesc}{quote_plus}{string\optional{, safe}} +Like \function{quote()}, but also replaces spaces by plus signs, as +required for quoting HTML form values. Plus signs in the original +string are escaped unless they are included in \var{safe}. It also +does not have \var{safe} default to \code{'/'}. +\end{funcdesc} + +\begin{funcdesc}{unquote}{string} +Replace \samp{\%xx} escapes by their single-character equivalent. + +Example: \code{unquote('/\%7Econnolly/')} yields \code{'/\~{}connolly/'}. +\end{funcdesc} + +\begin{funcdesc}{unquote_plus}{string} +Like \function{unquote()}, but also replaces plus signs by spaces, as +required for unquoting HTML form values. +\end{funcdesc} + +\begin{funcdesc}{urlencode}{query\optional{, doseq}} +Convert a mapping object or a sequence of two-element tuples to a +``url-encoded'' string, suitable to pass to +\function{urlopen()} above as the optional \var{data} argument. This +is useful to pass a dictionary of form fields to a \code{POST} +request. The resulting string is a series of +\code{\var{key}=\var{value}} pairs separated by \character{\&} +characters, where both \var{key} and \var{value} are quoted using +\function{quote_plus()} above. If the optional parameter \var{doseq} is +present and evaluates to true, individual \code{\var{key}=\var{value}} pairs +are generated for each element of the sequence. +When a sequence of two-element tuples is used as the \var{query} argument, +the first element of each tuple is a key and the second is a value. The +order of parameters in the encoded string will match the order of parameter +tuples in the sequence. +The \refmodule{cgi} module provides the functions +\function{parse_qs()} and \function{parse_qsl()} which are used to +parse query strings into Python data structures. +\end{funcdesc} + +\begin{funcdesc}{pathname2url}{path} +Convert the pathname \var{path} from the local syntax for a path to +the form used in the path component of a URL. This does not produce a +complete URL. The return value will already be quoted using the +\function{quote()} function. +\end{funcdesc} + +\begin{funcdesc}{url2pathname}{path} +Convert the path component \var{path} from an encoded URL to the local +syntax for a path. This does not accept a complete URL. This +function uses \function{unquote()} to decode \var{path}. +\end{funcdesc} + +\begin{classdesc}{URLopener}{\optional{proxies\optional{, **x509}}} +Base class for opening and reading URLs. Unless you need to support +opening objects using schemes other than \file{http:}, \file{ftp:}, +\file{gopher:} or \file{file:}, you probably want to use +\class{FancyURLopener}. + +By default, the \class{URLopener} class sends a +\mailheader{User-Agent} header of \samp{urllib/\var{VVV}}, where +\var{VVV} is the \module{urllib} version number. Applications can +define their own \mailheader{User-Agent} header by subclassing +\class{URLopener} or \class{FancyURLopener} and setting the class +attribute \member{version} to an appropriate string value in the +subclass definition. + +The optional \var{proxies} parameter should be a dictionary mapping +scheme names to proxy URLs, where an empty dictionary turns proxies +off completely. Its default value is \code{None}, in which case +environmental proxy settings will be used if present, as discussed in +the definition of \function{urlopen()}, above. + +Additional keyword parameters, collected in \var{x509}, may be used for +authentication of the client when using the \file{https:} scheme. The keywords +\var{key_file} and \var{cert_file} are supported to provide an +SSL key and certificate; both are needed to support client authentication. + +\class{URLopener} objects will raise an \exception{IOError} exception +if the server returns an error code. +\end{classdesc} + +\begin{classdesc}{FancyURLopener}{...} +\class{FancyURLopener} subclasses \class{URLopener} providing default +handling for the following HTTP response codes: 301, 302, 303, 307 and +401. For the 30x response codes listed above, the +\mailheader{Location} header is used to fetch the actual URL. For 401 +response codes (authentication required), basic HTTP authentication is +performed. For the 30x response codes, recursion is bounded by the +value of the \var{maxtries} attribute, which defaults to 10. + +For all other response codes, the method \method{http_error_default()} +is called which you can override in subclasses to handle the error +appropriately. + +\note{According to the letter of \rfc{2616}, 301 and 302 responses to + POST requests must not be automatically redirected without + confirmation by the user. In reality, browsers do allow automatic + redirection of these responses, changing the POST to a GET, and + \module{urllib} reproduces this behaviour.} + +The parameters to the constructor are the same as those for +\class{URLopener}. + +\note{When performing basic authentication, a +\class{FancyURLopener} instance calls its +\method{prompt_user_passwd()} method. The default implementation asks +the users for the required information on the controlling terminal. A +subclass may override this method to support more appropriate behavior +if needed.} +\end{classdesc} + +\begin{excclassdesc}{ContentTooShortError}{msg\optional{, content}} +This exception is raised when the \function{urlretrieve()} function +detects that the amount of the downloaded data is less than the +expected amount (given by the \var{Content-Length} header). The +\member{content} attribute stores the downloaded (and supposedly +truncated) data. +\versionadded{2.5} +\end{excclassdesc} + +Restrictions: + +\begin{itemize} + +\item +Currently, only the following protocols are supported: HTTP, (versions +0.9 and 1.0), Gopher (but not Gopher-+), FTP, and local files. +\indexii{HTTP}{protocol} +\indexii{Gopher}{protocol} +\indexii{FTP}{protocol} + +\item +The caching feature of \function{urlretrieve()} has been disabled +until I find the time to hack proper processing of Expiration time +headers. + +\item +There should be a function to query whether a particular URL is in +the cache. + +\item +For backward compatibility, if a URL appears to point to a local file +but the file can't be opened, the URL is re-interpreted using the FTP +protocol. This can sometimes cause confusing error messages. + +\item +The \function{urlopen()} and \function{urlretrieve()} functions can +cause arbitrarily long delays while waiting for a network connection +to be set up. This means that it is difficult to build an interactive +Web client using these functions without using threads. + +\item +The data returned by \function{urlopen()} or \function{urlretrieve()} +is the raw data returned by the server. This may be binary data +(such as an image), plain text or (for example) HTML\index{HTML}. The +HTTP\indexii{HTTP}{protocol} protocol provides type information in the +reply header, which can be inspected by looking at the +\mailheader{Content-Type} header. For the +Gopher\indexii{Gopher}{protocol} protocol, type information is encoded +in the URL; there is currently no easy way to extract it. If the +returned data is HTML, you can use the module +\refmodule{htmllib}\refstmodindex{htmllib} to parse it. + +\item +The code handling the FTP\index{FTP} protocol cannot differentiate +between a file and a directory. This can lead to unexpected behavior +when attempting to read a URL that points to a file that is not +accessible. If the URL ends in a \code{/}, it is assumed to refer to +a directory and will be handled accordingly. But if an attempt to +read a file leads to a 550 error (meaning the URL cannot be found or +is not accessible, often for permission reasons), then the path is +treated as a directory in order to handle the case when a directory is +specified by a URL but the trailing \code{/} has been left off. This can +cause misleading results when you try to fetch a file whose read +permissions make it inaccessible; the FTP code will try to read it, +fail with a 550 error, and then perform a directory listing for the +unreadable file. If fine-grained control is needed, consider using the +\module{ftplib} module, subclassing \class{FancyURLOpener}, or changing +\var{_urlopener} to meet your needs. + +\item +This module does not support the use of proxies which require +authentication. This may be implemented in the future. + +\item +Although the \module{urllib} module contains (undocumented) routines +to parse and unparse URL strings, the recommended interface for URL +manipulation is in module \refmodule{urlparse}\refstmodindex{urlparse}. + +\end{itemize} + + +\subsection{URLopener Objects \label{urlopener-objs}} +\sectionauthor{Skip Montanaro}{skip@mojam.com} + +\class{URLopener} and \class{FancyURLopener} objects have the +following attributes. + +\begin{methoddesc}[URLopener]{open}{fullurl\optional{, data}} +Open \var{fullurl} using the appropriate protocol. This method sets +up cache and proxy information, then calls the appropriate open method with +its input arguments. If the scheme is not recognized, +\method{open_unknown()} is called. The \var{data} argument +has the same meaning as the \var{data} argument of \function{urlopen()}. +\end{methoddesc} + +\begin{methoddesc}[URLopener]{open_unknown}{fullurl\optional{, data}} +Overridable interface to open unknown URL types. +\end{methoddesc} + +\begin{methoddesc}[URLopener]{retrieve}{url\optional{, + filename\optional{, + reporthook\optional{, data}}}} +Retrieves the contents of \var{url} and places it in \var{filename}. The +return value is a tuple consisting of a local filename and either a +\class{mimetools.Message} object containing the response headers (for remote +URLs) or \code{None} (for local URLs). The caller must then open and read the +contents of \var{filename}. If \var{filename} is not given and the URL +refers to a local file, the input filename is returned. If the URL is +non-local and \var{filename} is not given, the filename is the output of +\function{tempfile.mktemp()} with a suffix that matches the suffix of the last +path component of the input URL. If \var{reporthook} is given, it must be +a function accepting three numeric parameters. It will be called after each +chunk of data is read from the network. \var{reporthook} is ignored for +local URLs. + +If the \var{url} uses the \file{http:} scheme identifier, the optional +\var{data} argument may be given to specify a \code{POST} request +(normally the request type is \code{GET}). The \var{data} argument +must in standard \mimetype{application/x-www-form-urlencoded} format; +see the \function{urlencode()} function below. +\end{methoddesc} + +\begin{memberdesc}[URLopener]{version} +Variable that specifies the user agent of the opener object. To get +\refmodule{urllib} to tell servers that it is a particular user agent, +set this in a subclass as a class variable or in the constructor +before calling the base constructor. +\end{memberdesc} + +The \class{FancyURLopener} class offers one additional method that +should be overloaded to provide the appropriate behavior: + +\begin{methoddesc}[FancyURLopener]{prompt_user_passwd}{host, realm} +Return information needed to authenticate the user at the given host +in the specified security realm. The return value should be a tuple, +\code{(\var{user}, \var{password})}, which can be used for basic +authentication. + +The implementation prompts for this information on the terminal; an +application should override this method to use an appropriate +interaction model in the local environment. +\end{methoddesc} + + +\subsection{Examples} +\nodename{Urllib Examples} + +Here is an example session that uses the \samp{GET} method to retrieve +a URL containing parameters: + +\begin{verbatim} +>>> import urllib +>>> params = urllib.urlencode({'spam': 1, 'eggs': 2, 'bacon': 0}) +>>> f = urllib.urlopen("http://www.musi-cal.com/cgi-bin/query?%s" % params) +>>> print f.read() +\end{verbatim} + +The following example uses the \samp{POST} method instead: + +\begin{verbatim} +>>> import urllib +>>> params = urllib.urlencode({'spam': 1, 'eggs': 2, 'bacon': 0}) +>>> f = urllib.urlopen("http://www.musi-cal.com/cgi-bin/query", params) +>>> print f.read() +\end{verbatim} + +The following example uses an explicitly specified HTTP proxy, +overriding environment settings: + +\begin{verbatim} +>>> import urllib +>>> proxies = {'http': 'http://proxy.example.com:8080/'} +>>> opener = urllib.FancyURLopener(proxies) +>>> f = opener.open("http://www.python.org") +>>> f.read() +\end{verbatim} + +The following example uses no proxies at all, overriding environment +settings: + +\begin{verbatim} +>>> import urllib +>>> opener = urllib.FancyURLopener({}) +>>> f = opener.open("http://www.python.org/") +>>> f.read() +\end{verbatim} |