summaryrefslogtreecommitdiff
path: root/sys/src/cmd/python/Doc/lib/liburllib.tex
diff options
context:
space:
mode:
authorcinap_lenrek <cinap_lenrek@localhost>2011-05-03 11:25:13 +0000
committercinap_lenrek <cinap_lenrek@localhost>2011-05-03 11:25:13 +0000
commit458120dd40db6b4df55a4e96b650e16798ef06a0 (patch)
tree8f82685be24fef97e715c6f5ca4c68d34d5074ee /sys/src/cmd/python/Doc/lib/liburllib.tex
parent3a742c699f6806c1145aea5149bf15de15a0afd7 (diff)
add hg and python
Diffstat (limited to 'sys/src/cmd/python/Doc/lib/liburllib.tex')
-rw-r--r--sys/src/cmd/python/Doc/lib/liburllib.tex497
1 files changed, 497 insertions, 0 deletions
diff --git a/sys/src/cmd/python/Doc/lib/liburllib.tex b/sys/src/cmd/python/Doc/lib/liburllib.tex
new file mode 100644
index 000000000..75ee310d5
--- /dev/null
+++ b/sys/src/cmd/python/Doc/lib/liburllib.tex
@@ -0,0 +1,497 @@
+\section{\module{urllib} ---
+ Open arbitrary resources by URL}
+
+\declaremodule{standard}{urllib}
+\modulesynopsis{Open an arbitrary network resource by URL (requires sockets).}
+
+\index{WWW}
+\index{World Wide Web}
+\index{URL}
+
+
+This module provides a high-level interface for fetching data across
+the World Wide Web. In particular, the \function{urlopen()} function
+is similar to the built-in function \function{open()}, but accepts
+Universal Resource Locators (URLs) instead of filenames. Some
+restrictions apply --- it can only open URLs for reading, and no seek
+operations are available.
+
+It defines the following public functions:
+
+\begin{funcdesc}{urlopen}{url\optional{, data\optional{, proxies}}}
+Open a network object denoted by a URL for reading. If the URL does
+not have a scheme identifier, or if it has \file{file:} as its scheme
+identifier, this opens a local file (without universal newlines);
+otherwise it opens a socket to a server somewhere on the network. If
+the connection cannot be made
+the \exception{IOError} exception is raised. If all went well, a
+file-like object is returned. This supports the following methods:
+\method{read()}, \method{readline()}, \method{readlines()}, \method{fileno()},
+\method{close()}, \method{info()} and \method{geturl()}. It also has
+proper support for the iterator protocol.
+One caveat: the \method{read()} method, if the size argument is
+omitted or negative, may not read until the end of the data stream;
+there is no good way to determine that the entire stream from a socket
+has been read in the general case.
+
+Except for the \method{info()} and \method{geturl()} methods,
+these methods have the same interface as for
+file objects --- see section \ref{bltin-file-objects} in this
+manual. (It is not a built-in file object, however, so it can't be
+used at those few places where a true built-in file object is
+required.)
+
+The \method{info()} method returns an instance of the class
+\class{mimetools.Message} containing meta-information associated
+with the URL. When the method is HTTP, these headers are those
+returned by the server at the head of the retrieved HTML page
+(including Content-Length and Content-Type). When the method is FTP,
+a Content-Length header will be present if (as is now usual) the
+server passed back a file length in response to the FTP retrieval
+request. A Content-Type header will be present if the MIME type can
+be guessed. When the method is local-file, returned headers will include
+a Date representing the file's last-modified time, a Content-Length
+giving file size, and a Content-Type containing a guess at the file's
+type. See also the description of the
+\refmodule{mimetools}\refstmodindex{mimetools} module.
+
+The \method{geturl()} method returns the real URL of the page. In
+some cases, the HTTP server redirects a client to another URL. The
+\function{urlopen()} function handles this transparently, but in some
+cases the caller needs to know which URL the client was redirected
+to. The \method{geturl()} method can be used to get at this
+redirected URL.
+
+If the \var{url} uses the \file{http:} scheme identifier, the optional
+\var{data} argument may be given to specify a \code{POST} request
+(normally the request type is \code{GET}). The \var{data} argument
+must be in standard \mimetype{application/x-www-form-urlencoded} format;
+see the \function{urlencode()} function below.
+
+The \function{urlopen()} function works transparently with proxies
+which do not require authentication. In a \UNIX{} or Windows
+environment, set the \envvar{http_proxy}, \envvar{ftp_proxy} or
+\envvar{gopher_proxy} environment variables to a URL that identifies
+the proxy server before starting the Python interpreter. For example
+(the \character{\%} is the command prompt):
+
+\begin{verbatim}
+% http_proxy="http://www.someproxy.com:3128"
+% export http_proxy
+% python
+...
+\end{verbatim}
+
+In a Windows environment, if no proxy environment variables are set,
+proxy settings are obtained from the registry's Internet Settings
+section.
+
+In a Macintosh environment, \function{urlopen()} will retrieve proxy
+information from Internet\index{Internet Config} Config.
+
+Alternatively, the optional \var{proxies} argument may be used to
+explicitly specify proxies. It must be a dictionary mapping scheme
+names to proxy URLs, where an empty dictionary causes no proxies to be
+used, and \code{None} (the default value) causes environmental proxy
+settings to be used as discussed above. For example:
+
+\begin{verbatim}
+# Use http://www.someproxy.com:3128 for http proxying
+proxies = {'http': 'http://www.someproxy.com:3128'}
+filehandle = urllib.urlopen(some_url, proxies=proxies)
+# Don't use any proxies
+filehandle = urllib.urlopen(some_url, proxies={})
+# Use proxies from environment - both versions are equivalent
+filehandle = urllib.urlopen(some_url, proxies=None)
+filehandle = urllib.urlopen(some_url)
+\end{verbatim}
+
+The \function{urlopen()} function does not support explicit proxy
+specification. If you need to override environmental proxy settings,
+use \class{URLopener}, or a subclass such as \class{FancyURLopener}.
+
+Proxies which require authentication for use are not currently
+supported; this is considered an implementation limitation.
+
+\versionchanged[Added the \var{proxies} support]{2.3}
+\end{funcdesc}
+
+\begin{funcdesc}{urlretrieve}{url\optional{, filename\optional{,
+ reporthook\optional{, data}}}}
+Copy a network object denoted by a URL to a local file, if necessary.
+If the URL points to a local file, or a valid cached copy of the
+object exists, the object is not copied. Return a tuple
+\code{(\var{filename}, \var{headers})} where \var{filename} is the
+local file name under which the object can be found, and \var{headers}
+is whatever the \method{info()} method of the object returned by
+\function{urlopen()} returned (for a remote object, possibly cached).
+Exceptions are the same as for \function{urlopen()}.
+
+The second argument, if present, specifies the file location to copy
+to (if absent, the location will be a tempfile with a generated name).
+The third argument, if present, is a hook function that will be called
+once on establishment of the network connection and once after each
+block read thereafter. The hook will be passed three arguments; a
+count of blocks transferred so far, a block size in bytes, and the
+total size of the file. The third argument may be \code{-1} on older
+FTP servers which do not return a file size in response to a retrieval
+request.
+
+If the \var{url} uses the \file{http:} scheme identifier, the optional
+\var{data} argument may be given to specify a \code{POST} request
+(normally the request type is \code{GET}). The \var{data} argument
+must in standard \mimetype{application/x-www-form-urlencoded} format;
+see the \function{urlencode()} function below.
+
+\versionchanged[
+\function{urlretrieve()} will raise \exception{ContentTooShortError}
+when it detects that the amount of data available
+was less than the expected amount (which is the size reported by a
+\var{Content-Length} header). This can occur, for example, when the
+download is interrupted.
+
+The \var{Content-Length} is treated as a lower bound: if there's more data
+to read, urlretrieve reads more data, but if less data is available,
+it raises the exception.
+
+You can still retrieve the downloaded data in this case, it is stored
+in the \member{content} attribute of the exception instance.
+
+If no \var{Content-Length} header was supplied, urlretrieve can
+not check the size of the data it has downloaded, and just returns it.
+In this case you just have to assume that the download was successful]{2.5}
+
+\end{funcdesc}
+
+\begin{datadesc}{_urlopener}
+The public functions \function{urlopen()} and
+\function{urlretrieve()} create an instance of the
+\class{FancyURLopener} class and use it to perform their requested
+actions. To override this functionality, programmers can create a
+subclass of \class{URLopener} or \class{FancyURLopener}, then assign
+an instance of that class to the
+\code{urllib._urlopener} variable before calling the desired function.
+For example, applications may want to specify a different
+\mailheader{User-Agent} header than \class{URLopener} defines. This
+can be accomplished with the following code:
+
+\begin{verbatim}
+import urllib
+
+class AppURLopener(urllib.FancyURLopener):
+ version = "App/1.7"
+
+urllib._urlopener = AppURLopener()
+\end{verbatim}
+\end{datadesc}
+
+\begin{funcdesc}{urlcleanup}{}
+Clear the cache that may have been built up by previous calls to
+\function{urlretrieve()}.
+\end{funcdesc}
+
+\begin{funcdesc}{quote}{string\optional{, safe}}
+Replace special characters in \var{string} using the \samp{\%xx} escape.
+Letters, digits, and the characters \character{_.-} are never quoted.
+The optional \var{safe} parameter specifies additional characters
+that should not be quoted --- its default value is \code{'/'}.
+
+Example: \code{quote('/\~{}connolly/')} yields \code{'/\%7econnolly/'}.
+\end{funcdesc}
+
+\begin{funcdesc}{quote_plus}{string\optional{, safe}}
+Like \function{quote()}, but also replaces spaces by plus signs, as
+required for quoting HTML form values. Plus signs in the original
+string are escaped unless they are included in \var{safe}. It also
+does not have \var{safe} default to \code{'/'}.
+\end{funcdesc}
+
+\begin{funcdesc}{unquote}{string}
+Replace \samp{\%xx} escapes by their single-character equivalent.
+
+Example: \code{unquote('/\%7Econnolly/')} yields \code{'/\~{}connolly/'}.
+\end{funcdesc}
+
+\begin{funcdesc}{unquote_plus}{string}
+Like \function{unquote()}, but also replaces plus signs by spaces, as
+required for unquoting HTML form values.
+\end{funcdesc}
+
+\begin{funcdesc}{urlencode}{query\optional{, doseq}}
+Convert a mapping object or a sequence of two-element tuples to a
+``url-encoded'' string, suitable to pass to
+\function{urlopen()} above as the optional \var{data} argument. This
+is useful to pass a dictionary of form fields to a \code{POST}
+request. The resulting string is a series of
+\code{\var{key}=\var{value}} pairs separated by \character{\&}
+characters, where both \var{key} and \var{value} are quoted using
+\function{quote_plus()} above. If the optional parameter \var{doseq} is
+present and evaluates to true, individual \code{\var{key}=\var{value}} pairs
+are generated for each element of the sequence.
+When a sequence of two-element tuples is used as the \var{query} argument,
+the first element of each tuple is a key and the second is a value. The
+order of parameters in the encoded string will match the order of parameter
+tuples in the sequence.
+The \refmodule{cgi} module provides the functions
+\function{parse_qs()} and \function{parse_qsl()} which are used to
+parse query strings into Python data structures.
+\end{funcdesc}
+
+\begin{funcdesc}{pathname2url}{path}
+Convert the pathname \var{path} from the local syntax for a path to
+the form used in the path component of a URL. This does not produce a
+complete URL. The return value will already be quoted using the
+\function{quote()} function.
+\end{funcdesc}
+
+\begin{funcdesc}{url2pathname}{path}
+Convert the path component \var{path} from an encoded URL to the local
+syntax for a path. This does not accept a complete URL. This
+function uses \function{unquote()} to decode \var{path}.
+\end{funcdesc}
+
+\begin{classdesc}{URLopener}{\optional{proxies\optional{, **x509}}}
+Base class for opening and reading URLs. Unless you need to support
+opening objects using schemes other than \file{http:}, \file{ftp:},
+\file{gopher:} or \file{file:}, you probably want to use
+\class{FancyURLopener}.
+
+By default, the \class{URLopener} class sends a
+\mailheader{User-Agent} header of \samp{urllib/\var{VVV}}, where
+\var{VVV} is the \module{urllib} version number. Applications can
+define their own \mailheader{User-Agent} header by subclassing
+\class{URLopener} or \class{FancyURLopener} and setting the class
+attribute \member{version} to an appropriate string value in the
+subclass definition.
+
+The optional \var{proxies} parameter should be a dictionary mapping
+scheme names to proxy URLs, where an empty dictionary turns proxies
+off completely. Its default value is \code{None}, in which case
+environmental proxy settings will be used if present, as discussed in
+the definition of \function{urlopen()}, above.
+
+Additional keyword parameters, collected in \var{x509}, may be used for
+authentication of the client when using the \file{https:} scheme. The keywords
+\var{key_file} and \var{cert_file} are supported to provide an
+SSL key and certificate; both are needed to support client authentication.
+
+\class{URLopener} objects will raise an \exception{IOError} exception
+if the server returns an error code.
+\end{classdesc}
+
+\begin{classdesc}{FancyURLopener}{...}
+\class{FancyURLopener} subclasses \class{URLopener} providing default
+handling for the following HTTP response codes: 301, 302, 303, 307 and
+401. For the 30x response codes listed above, the
+\mailheader{Location} header is used to fetch the actual URL. For 401
+response codes (authentication required), basic HTTP authentication is
+performed. For the 30x response codes, recursion is bounded by the
+value of the \var{maxtries} attribute, which defaults to 10.
+
+For all other response codes, the method \method{http_error_default()}
+is called which you can override in subclasses to handle the error
+appropriately.
+
+\note{According to the letter of \rfc{2616}, 301 and 302 responses to
+ POST requests must not be automatically redirected without
+ confirmation by the user. In reality, browsers do allow automatic
+ redirection of these responses, changing the POST to a GET, and
+ \module{urllib} reproduces this behaviour.}
+
+The parameters to the constructor are the same as those for
+\class{URLopener}.
+
+\note{When performing basic authentication, a
+\class{FancyURLopener} instance calls its
+\method{prompt_user_passwd()} method. The default implementation asks
+the users for the required information on the controlling terminal. A
+subclass may override this method to support more appropriate behavior
+if needed.}
+\end{classdesc}
+
+\begin{excclassdesc}{ContentTooShortError}{msg\optional{, content}}
+This exception is raised when the \function{urlretrieve()} function
+detects that the amount of the downloaded data is less than the
+expected amount (given by the \var{Content-Length} header). The
+\member{content} attribute stores the downloaded (and supposedly
+truncated) data.
+\versionadded{2.5}
+\end{excclassdesc}
+
+Restrictions:
+
+\begin{itemize}
+
+\item
+Currently, only the following protocols are supported: HTTP, (versions
+0.9 and 1.0), Gopher (but not Gopher-+), FTP, and local files.
+\indexii{HTTP}{protocol}
+\indexii{Gopher}{protocol}
+\indexii{FTP}{protocol}
+
+\item
+The caching feature of \function{urlretrieve()} has been disabled
+until I find the time to hack proper processing of Expiration time
+headers.
+
+\item
+There should be a function to query whether a particular URL is in
+the cache.
+
+\item
+For backward compatibility, if a URL appears to point to a local file
+but the file can't be opened, the URL is re-interpreted using the FTP
+protocol. This can sometimes cause confusing error messages.
+
+\item
+The \function{urlopen()} and \function{urlretrieve()} functions can
+cause arbitrarily long delays while waiting for a network connection
+to be set up. This means that it is difficult to build an interactive
+Web client using these functions without using threads.
+
+\item
+The data returned by \function{urlopen()} or \function{urlretrieve()}
+is the raw data returned by the server. This may be binary data
+(such as an image), plain text or (for example) HTML\index{HTML}. The
+HTTP\indexii{HTTP}{protocol} protocol provides type information in the
+reply header, which can be inspected by looking at the
+\mailheader{Content-Type} header. For the
+Gopher\indexii{Gopher}{protocol} protocol, type information is encoded
+in the URL; there is currently no easy way to extract it. If the
+returned data is HTML, you can use the module
+\refmodule{htmllib}\refstmodindex{htmllib} to parse it.
+
+\item
+The code handling the FTP\index{FTP} protocol cannot differentiate
+between a file and a directory. This can lead to unexpected behavior
+when attempting to read a URL that points to a file that is not
+accessible. If the URL ends in a \code{/}, it is assumed to refer to
+a directory and will be handled accordingly. But if an attempt to
+read a file leads to a 550 error (meaning the URL cannot be found or
+is not accessible, often for permission reasons), then the path is
+treated as a directory in order to handle the case when a directory is
+specified by a URL but the trailing \code{/} has been left off. This can
+cause misleading results when you try to fetch a file whose read
+permissions make it inaccessible; the FTP code will try to read it,
+fail with a 550 error, and then perform a directory listing for the
+unreadable file. If fine-grained control is needed, consider using the
+\module{ftplib} module, subclassing \class{FancyURLOpener}, or changing
+\var{_urlopener} to meet your needs.
+
+\item
+This module does not support the use of proxies which require
+authentication. This may be implemented in the future.
+
+\item
+Although the \module{urllib} module contains (undocumented) routines
+to parse and unparse URL strings, the recommended interface for URL
+manipulation is in module \refmodule{urlparse}\refstmodindex{urlparse}.
+
+\end{itemize}
+
+
+\subsection{URLopener Objects \label{urlopener-objs}}
+\sectionauthor{Skip Montanaro}{skip@mojam.com}
+
+\class{URLopener} and \class{FancyURLopener} objects have the
+following attributes.
+
+\begin{methoddesc}[URLopener]{open}{fullurl\optional{, data}}
+Open \var{fullurl} using the appropriate protocol. This method sets
+up cache and proxy information, then calls the appropriate open method with
+its input arguments. If the scheme is not recognized,
+\method{open_unknown()} is called. The \var{data} argument
+has the same meaning as the \var{data} argument of \function{urlopen()}.
+\end{methoddesc}
+
+\begin{methoddesc}[URLopener]{open_unknown}{fullurl\optional{, data}}
+Overridable interface to open unknown URL types.
+\end{methoddesc}
+
+\begin{methoddesc}[URLopener]{retrieve}{url\optional{,
+ filename\optional{,
+ reporthook\optional{, data}}}}
+Retrieves the contents of \var{url} and places it in \var{filename}. The
+return value is a tuple consisting of a local filename and either a
+\class{mimetools.Message} object containing the response headers (for remote
+URLs) or \code{None} (for local URLs). The caller must then open and read the
+contents of \var{filename}. If \var{filename} is not given and the URL
+refers to a local file, the input filename is returned. If the URL is
+non-local and \var{filename} is not given, the filename is the output of
+\function{tempfile.mktemp()} with a suffix that matches the suffix of the last
+path component of the input URL. If \var{reporthook} is given, it must be
+a function accepting three numeric parameters. It will be called after each
+chunk of data is read from the network. \var{reporthook} is ignored for
+local URLs.
+
+If the \var{url} uses the \file{http:} scheme identifier, the optional
+\var{data} argument may be given to specify a \code{POST} request
+(normally the request type is \code{GET}). The \var{data} argument
+must in standard \mimetype{application/x-www-form-urlencoded} format;
+see the \function{urlencode()} function below.
+\end{methoddesc}
+
+\begin{memberdesc}[URLopener]{version}
+Variable that specifies the user agent of the opener object. To get
+\refmodule{urllib} to tell servers that it is a particular user agent,
+set this in a subclass as a class variable or in the constructor
+before calling the base constructor.
+\end{memberdesc}
+
+The \class{FancyURLopener} class offers one additional method that
+should be overloaded to provide the appropriate behavior:
+
+\begin{methoddesc}[FancyURLopener]{prompt_user_passwd}{host, realm}
+Return information needed to authenticate the user at the given host
+in the specified security realm. The return value should be a tuple,
+\code{(\var{user}, \var{password})}, which can be used for basic
+authentication.
+
+The implementation prompts for this information on the terminal; an
+application should override this method to use an appropriate
+interaction model in the local environment.
+\end{methoddesc}
+
+
+\subsection{Examples}
+\nodename{Urllib Examples}
+
+Here is an example session that uses the \samp{GET} method to retrieve
+a URL containing parameters:
+
+\begin{verbatim}
+>>> import urllib
+>>> params = urllib.urlencode({'spam': 1, 'eggs': 2, 'bacon': 0})
+>>> f = urllib.urlopen("http://www.musi-cal.com/cgi-bin/query?%s" % params)
+>>> print f.read()
+\end{verbatim}
+
+The following example uses the \samp{POST} method instead:
+
+\begin{verbatim}
+>>> import urllib
+>>> params = urllib.urlencode({'spam': 1, 'eggs': 2, 'bacon': 0})
+>>> f = urllib.urlopen("http://www.musi-cal.com/cgi-bin/query", params)
+>>> print f.read()
+\end{verbatim}
+
+The following example uses an explicitly specified HTTP proxy,
+overriding environment settings:
+
+\begin{verbatim}
+>>> import urllib
+>>> proxies = {'http': 'http://proxy.example.com:8080/'}
+>>> opener = urllib.FancyURLopener(proxies)
+>>> f = opener.open("http://www.python.org")
+>>> f.read()
+\end{verbatim}
+
+The following example uses no proxies at all, overriding environment
+settings:
+
+\begin{verbatim}
+>>> import urllib
+>>> opener = urllib.FancyURLopener({})
+>>> f = opener.open("http://www.python.org/")
+>>> f.read()
+\end{verbatim}