add hg and python

1 files changed, 602 insertions, 0 deletions

diff --git a/sys/src/cmd/python/Doc/lib/libcgi.tex b/sys/src/cmd/python/Doc/lib/libcgi.tex
new file mode 100644
index 000000000..1dd7e03f3
--- /dev/null
+++ b/sys/src/cmd/python/Doc/lib/libcgi.tex
@@ -0,0 +1,602 @@
+\section{\module{cgi} ---
+         Common Gateway Interface support.}
+\declaremodule{standard}{cgi}
+
+\modulesynopsis{Common Gateway Interface support, used to interpret
+forms in server-side scripts.}
+
+\indexii{WWW}{server}
+\indexii{CGI}{protocol}
+\indexii{HTTP}{protocol}
+\indexii{MIME}{headers}
+\index{URL}
+
+
+Support module for Common Gateway Interface (CGI) scripts.%
+\index{Common Gateway Interface}
+
+This module defines a number of utilities for use by CGI scripts
+written in Python.
+
+\subsection{Introduction}
+\nodename{cgi-intro}
+
+A CGI script is invoked by an HTTP server, usually to process user
+input submitted through an HTML \code{<FORM>} or \code{<ISINDEX>} element.
+
+Most often, CGI scripts live in the server's special \file{cgi-bin}
+directory.  The HTTP server places all sorts of information about the
+request (such as the client's hostname, the requested URL, the query
+string, and lots of other goodies) in the script's shell environment,
+executes the script, and sends the script's output back to the client.
+
+The script's input is connected to the client too, and sometimes the
+form data is read this way; at other times the form data is passed via
+the ``query string'' part of the URL.  This module is intended
+to take care of the different cases and provide a simpler interface to
+the Python script.  It also provides a number of utilities that help
+in debugging scripts, and the latest addition is support for file
+uploads from a form (if your browser supports it).
+
+The output of a CGI script should consist of two sections, separated
+by a blank line.  The first section contains a number of headers,
+telling the client what kind of data is following.  Python code to
+generate a minimal header section looks like this:
+
+\begin{verbatim}
+print "Content-Type: text/html"     # HTML is following
+print                               # blank line, end of headers
+\end{verbatim}
+
+The second section is usually HTML, which allows the client software
+to display nicely formatted text with header, in-line images, etc.
+Here's Python code that prints a simple piece of HTML:
+
+\begin{verbatim}
+print "<TITLE>CGI script output</TITLE>"
+print "<H1>This is my first CGI script</H1>"
+print "Hello, world!"
+\end{verbatim}
+
+\subsection{Using the cgi module}
+\nodename{Using the cgi module}
+
+Begin by writing \samp{import cgi}.  Do not use \samp{from cgi import
+*} --- the module defines all sorts of names for its own use or for
+backward compatibility that you don't want in your namespace.
+
+When you write a new script, consider adding the line:
+
+\begin{verbatim}
+import cgitb; cgitb.enable()
+\end{verbatim}
+
+This activates a special exception handler that will display detailed
+reports in the Web browser if any errors occur.  If you'd rather not
+show the guts of your program to users of your script, you can have
+the reports saved to files instead, with a line like this:
+
+\begin{verbatim}
+import cgitb; cgitb.enable(display=0, logdir="/tmp")
+\end{verbatim}
+
+It's very helpful to use this feature during script development.
+The reports produced by \refmodule{cgitb} provide information that
+can save you a lot of time in tracking down bugs.  You can always
+remove the \code{cgitb} line later when you have tested your script
+and are confident that it works correctly.
+
+To get at submitted form data,
+it's best to use the \class{FieldStorage} class.  The other classes
+defined in this module are provided mostly for backward compatibility.
+Instantiate it exactly once, without arguments.  This reads the form
+contents from standard input or the environment (depending on the
+value of various environment variables set according to the CGI
+standard).  Since it may consume standard input, it should be
+instantiated only once.
+
+The \class{FieldStorage} instance can be indexed like a Python
+dictionary, and also supports the standard dictionary methods
+\method{has_key()} and \method{keys()}.  The built-in \function{len()}
+is also supported.  Form fields containing empty strings are ignored
+and do not appear in the dictionary; to keep such values, provide
+a true value for the optional \var{keep_blank_values} keyword
+parameter when creating the \class{FieldStorage} instance.
+
+For instance, the following code (which assumes that the 
+\mailheader{Content-Type} header and blank line have already been
+printed) checks that the fields \code{name} and \code{addr} are both
+set to a non-empty string:
+
+\begin{verbatim}
+form = cgi.FieldStorage()
+if not (form.has_key("name") and form.has_key("addr")):
+    print "<H1>Error</H1>"
+    print "Please fill in the name and addr fields."
+    return
+print "<p>name:", form["name"].value
+print "<p>addr:", form["addr"].value
+...further form processing here...
+\end{verbatim}
+
+Here the fields, accessed through \samp{form[\var{key}]}, are
+themselves instances of \class{FieldStorage} (or
+\class{MiniFieldStorage}, depending on the form encoding).
+The \member{value} attribute of the instance yields the string value
+of the field.  The \method{getvalue()} method returns this string value
+directly; it also accepts an optional second argument as a default to
+return if the requested key is not present.
+
+If the submitted form data contains more than one field with the same
+name, the object retrieved by \samp{form[\var{key}]} is not a
+\class{FieldStorage} or \class{MiniFieldStorage}
+instance but a list of such instances.  Similarly, in this situation,
+\samp{form.getvalue(\var{key})} would return a list of strings.
+If you expect this possibility
+(when your HTML form contains multiple fields with the same name), use
+the \function{getlist()} function, which always returns a list of values (so that you
+do not need to special-case the single item case).  For example, this
+code concatenates any number of username fields, separated by
+commas:
+
+\begin{verbatim}
+value = form.getlist("username")
+usernames = ",".join(value)
+\end{verbatim}
+
+If a field represents an uploaded file, accessing the value via the
+\member{value} attribute or the \function{getvalue()} method reads the
+entire file in memory as a string.  This may not be what you want.
+You can test for an uploaded file by testing either the \member{filename}
+attribute or the \member{file} attribute.  You can then read the data at
+leisure from the \member{file} attribute:
+
+\begin{verbatim}
+fileitem = form["userfile"]
+if fileitem.file:
+    # It's an uploaded file; count lines
+    linecount = 0
+    while 1:
+        line = fileitem.file.readline()
+        if not line: break
+        linecount = linecount + 1
+\end{verbatim}
+
+The file upload draft standard entertains the possibility of uploading
+multiple files from one field (using a recursive
+\mimetype{multipart/*} encoding).  When this occurs, the item will be
+a dictionary-like \class{FieldStorage} item.  This can be determined
+by testing its \member{type} attribute, which should be
+\mimetype{multipart/form-data} (or perhaps another MIME type matching
+\mimetype{multipart/*}).  In this case, it can be iterated over
+recursively just like the top-level form object.
+
+When a form is submitted in the ``old'' format (as the query string or
+as a single data part of type
+\mimetype{application/x-www-form-urlencoded}), the items will actually
+be instances of the class \class{MiniFieldStorage}.  In this case, the
+\member{list}, \member{file}, and \member{filename} attributes are
+always \code{None}.
+
+
+\subsection{Higher Level Interface}
+
+\versionadded{2.2}  % XXX: Is this true ? 
+
+The previous section explains how to read CGI form data using the
+\class{FieldStorage} class.  This section describes a higher level
+interface which was added to this class to allow one to do it in a
+more readable and intuitive way.  The interface doesn't make the
+techniques described in previous sections obsolete --- they are still
+useful to process file uploads efficiently, for example.
+
+The interface consists of two simple methods. Using the methods
+you can process form data in a generic way, without the need to worry
+whether only one or more values were posted under one name.
+
+In the previous section, you learned to write following code anytime
+you expected a user to post more than one value under one name:
+
+\begin{verbatim}
+item = form.getvalue("item")
+if isinstance(item, list):
+    # The user is requesting more than one item.
+else:
+    # The user is requesting only one item.
+\end{verbatim}
+
+This situation is common for example when a form contains a group of
+multiple checkboxes with the same name:
+
+\begin{verbatim}
+<input type="checkbox" name="item" value="1" />
+<input type="checkbox" name="item" value="2" />
+\end{verbatim}
+
+In most situations, however, there's only one form control with a
+particular name in a form and then you expect and need only one value
+associated with this name.  So you write a script containing for
+example this code:
+
+\begin{verbatim}
+user = form.getvalue("user").upper()
+\end{verbatim}
+
+The problem with the code is that you should never expect that a
+client will provide valid input to your scripts.  For example, if a
+curious user appends another \samp{user=foo} pair to the query string,
+then the script would crash, because in this situation the
+\code{getvalue("user")} method call returns a list instead of a
+string.  Calling the \method{toupper()} method on a list is not valid
+(since lists do not have a method of this name) and results in an
+\exception{AttributeError} exception.
+
+Therefore, the appropriate way to read form data values was to always
+use the code which checks whether the obtained value is a single value
+or a list of values.  That's annoying and leads to less readable
+scripts.
+
+A more convenient approach is to use the methods \method{getfirst()}
+and \method{getlist()} provided by this higher level interface.
+
+\begin{methoddesc}[FieldStorage]{getfirst}{name\optional{, default}}
+  This method always returns only one value associated with form field
+  \var{name}.  The method returns only the first value in case that
+  more values were posted under such name.  Please note that the order
+  in which the values are received may vary from browser to browser
+  and should not be counted on.\footnote{Note that some recent
+      versions of the HTML specification do state what order the
+      field values should be supplied in, but knowing whether a
+      request was received from a conforming browser, or even from a
+      browser at all, is tedious and error-prone.}  If no such form
+  field or value exists then the method returns the value specified by
+  the optional parameter \var{default}.  This parameter defaults to
+  \code{None} if not specified.
+\end{methoddesc}
+
+\begin{methoddesc}[FieldStorage]{getlist}{name}
+  This method always returns a list of values associated with form
+  field \var{name}.  The method returns an empty list if no such form
+  field or value exists for \var{name}.  It returns a list consisting
+  of one item if only one such value exists.
+\end{methoddesc}
+
+Using these methods you can write nice compact code:
+
+\begin{verbatim}
+import cgi
+form = cgi.FieldStorage()
+user = form.getfirst("user", "").upper()    # This way it's safe.
+for item in form.getlist("item"):
+    do_something(item)
+\end{verbatim}
+
+
+\subsection{Old classes}
+
+These classes, present in earlier versions of the \module{cgi} module,
+are still supported for backward compatibility.  New applications
+should use the \class{FieldStorage} class.
+
+\class{SvFormContentDict} stores single value form content as
+dictionary; it assumes each field name occurs in the form only once.
+
+\class{FormContentDict} stores multiple value form content as a
+dictionary (the form items are lists of values).  Useful if your form
+contains multiple fields with the same name.
+
+Other classes (\class{FormContent}, \class{InterpFormContentDict}) are
+present for backwards compatibility with really old applications only.
+If you still use these and would be inconvenienced when they
+disappeared from a next version of this module, drop me a note.
+
+
+\subsection{Functions}
+\nodename{Functions in cgi module}
+
+These are useful if you want more control, or if you want to employ
+some of the algorithms implemented in this module in other
+circumstances.
+
+\begin{funcdesc}{parse}{fp\optional{, keep_blank_values\optional{,
+                        strict_parsing}}}
+  Parse a query in the environment or from a file (the file defaults
+  to \code{sys.stdin}).  The \var{keep_blank_values} and
+  \var{strict_parsing} parameters are passed to \function{parse_qs()}
+  unchanged.
+\end{funcdesc}
+
+\begin{funcdesc}{parse_qs}{qs\optional{, keep_blank_values\optional{,
+                           strict_parsing}}}
+Parse a query string given as a string argument (data of type 
+\mimetype{application/x-www-form-urlencoded}).  Data are
+returned as a dictionary.  The dictionary keys are the unique query
+variable names and the values are lists of values for each name.
+
+The optional argument \var{keep_blank_values} is
+a flag indicating whether blank values in
+URL encoded queries should be treated as blank strings.  
+A true value indicates that blanks should be retained as 
+blank strings.  The default false value indicates that
+blank values are to be ignored and treated as if they were
+not included.
+
+The optional argument \var{strict_parsing} is a flag indicating what
+to do with parsing errors.  If false (the default), errors
+are silently ignored.  If true, errors raise a \exception{ValueError}
+exception.
+
+Use the \function{\refmodule{urllib}.urlencode()} function to convert
+such dictionaries into query strings.
+
+\end{funcdesc}
+
+\begin{funcdesc}{parse_qsl}{qs\optional{, keep_blank_values\optional{,
+                            strict_parsing}}}
+Parse a query string given as a string argument (data of type 
+\mimetype{application/x-www-form-urlencoded}).  Data are
+returned as a list of name, value pairs.
+
+The optional argument \var{keep_blank_values} is
+a flag indicating whether blank values in
+URL encoded queries should be treated as blank strings.  
+A true value indicates that blanks should be retained as 
+blank strings.  The default false value indicates that
+blank values are to be ignored and treated as if they were
+not included.
+
+The optional argument \var{strict_parsing} is a flag indicating what
+to do with parsing errors.  If false (the default), errors
+are silently ignored.  If true, errors raise a \exception{ValueError}
+exception.
+
+Use the \function{\refmodule{urllib}.urlencode()} function to convert
+such lists of pairs into query strings.
+\end{funcdesc}
+
+\begin{funcdesc}{parse_multipart}{fp, pdict}
+Parse input of type \mimetype{multipart/form-data} (for 
+file uploads).  Arguments are \var{fp} for the input file and
+\var{pdict} for a dictionary containing other parameters in
+the \mailheader{Content-Type} header.
+
+Returns a dictionary just like \function{parse_qs()} keys are the
+field names, each value is a list of values for that field.  This is
+easy to use but not much good if you are expecting megabytes to be
+uploaded --- in that case, use the \class{FieldStorage} class instead
+which is much more flexible.
+
+Note that this does not parse nested multipart parts --- use
+\class{FieldStorage} for that.
+\end{funcdesc}
+
+\begin{funcdesc}{parse_header}{string}
+Parse a MIME header (such as \mailheader{Content-Type}) into a main
+value and a dictionary of parameters.
+\end{funcdesc}
+
+\begin{funcdesc}{test}{}
+Robust test CGI script, usable as main program.
+Writes minimal HTTP headers and formats all information provided to
+the script in HTML form.
+\end{funcdesc}
+
+\begin{funcdesc}{print_environ}{}
+Format the shell environment in HTML.
+\end{funcdesc}
+
+\begin{funcdesc}{print_form}{form}
+Format a form in HTML.
+\end{funcdesc}
+
+\begin{funcdesc}{print_directory}{}
+Format the current directory in HTML.
+\end{funcdesc}
+
+\begin{funcdesc}{print_environ_usage}{}
+Print a list of useful (used by CGI) environment variables in
+HTML.
+\end{funcdesc}
+
+\begin{funcdesc}{escape}{s\optional{, quote}}
+Convert the characters
+\character{\&}, \character{<} and \character{>} in string \var{s} to
+HTML-safe sequences.  Use this if you need to display text that might
+contain such characters in HTML.  If the optional flag \var{quote} is
+true, the quotation mark character (\character{"}) is also translated;
+this helps for inclusion in an HTML attribute value, as in \code{<A
+HREF="...">}.  If the value to be quoted might include single- or
+double-quote characters, or both, consider using the
+\function{quoteattr()} function in the \refmodule{xml.sax.saxutils}
+module instead.
+\end{funcdesc}
+
+
+\subsection{Caring about security \label{cgi-security}}
+
+\indexii{CGI}{security}
+
+There's one important rule: if you invoke an external program (via the
+\function{os.system()} or \function{os.popen()} functions. or others
+with similar functionality), make very sure you don't pass arbitrary
+strings received from the client to the shell.  This is a well-known
+security hole whereby clever hackers anywhere on the Web can exploit a
+gullible CGI script to invoke arbitrary shell commands.  Even parts of
+the URL or field names cannot be trusted, since the request doesn't
+have to come from your form!
+
+To be on the safe side, if you must pass a string gotten from a form
+to a shell command, you should make sure the string contains only
+alphanumeric characters, dashes, underscores, and periods.
+
+
+\subsection{Installing your CGI script on a \UNIX\ system}
+
+Read the documentation for your HTTP server and check with your local
+system administrator to find the directory where CGI scripts should be
+installed; usually this is in a directory \file{cgi-bin} in the server tree.
+
+Make sure that your script is readable and executable by ``others''; the
+\UNIX{} file mode should be \code{0755} octal (use \samp{chmod 0755
+\var{filename}}).  Make sure that the first line of the script contains
+\code{\#!} starting in column 1 followed by the pathname of the Python
+interpreter, for instance:
+
+\begin{verbatim}
+#!/usr/local/bin/python
+\end{verbatim}
+
+Make sure the Python interpreter exists and is executable by ``others''.
+
+Make sure that any files your script needs to read or write are
+readable or writable, respectively, by ``others'' --- their mode
+should be \code{0644} for readable and \code{0666} for writable.  This
+is because, for security reasons, the HTTP server executes your script
+as user ``nobody'', without any special privileges.  It can only read
+(write, execute) files that everybody can read (write, execute).  The
+current directory at execution time is also different (it is usually
+the server's cgi-bin directory) and the set of environment variables
+is also different from what you get when you log in.  In particular, don't
+count on the shell's search path for executables (\envvar{PATH}) or
+the Python module search path (\envvar{PYTHONPATH}) to be set to
+anything interesting.
+
+If you need to load modules from a directory which is not on Python's
+default module search path, you can change the path in your script,
+before importing other modules.  For example:
+
+\begin{verbatim}
+import sys
+sys.path.insert(0, "/usr/home/joe/lib/python")
+sys.path.insert(0, "/usr/local/lib/python")
+\end{verbatim}
+
+(This way, the directory inserted last will be searched first!)
+
+Instructions for non-\UNIX{} systems will vary; check your HTTP server's
+documentation (it will usually have a section on CGI scripts).
+
+
+\subsection{Testing your CGI script}
+
+Unfortunately, a CGI script will generally not run when you try it
+from the command line, and a script that works perfectly from the
+command line may fail mysteriously when run from the server.  There's
+one reason why you should still test your script from the command
+line: if it contains a syntax error, the Python interpreter won't
+execute it at all, and the HTTP server will most likely send a cryptic
+error to the client.
+
+Assuming your script has no syntax errors, yet it does not work, you
+have no choice but to read the next section.
+
+
+\subsection{Debugging CGI scripts} \indexii{CGI}{debugging}
+
+First of all, check for trivial installation errors --- reading the
+section above on installing your CGI script carefully can save you a
+lot of time.  If you wonder whether you have understood the
+installation procedure correctly, try installing a copy of this module
+file (\file{cgi.py}) as a CGI script.  When invoked as a script, the file
+will dump its environment and the contents of the form in HTML form.
+Give it the right mode etc, and send it a request.  If it's installed
+in the standard \file{cgi-bin} directory, it should be possible to send it a
+request by entering a URL into your browser of the form:
+
+\begin{verbatim}
+http://yourhostname/cgi-bin/cgi.py?name=Joe+Blow&addr=At+Home
+\end{verbatim}
+
+If this gives an error of type 404, the server cannot find the script
+-- perhaps you need to install it in a different directory.  If it
+gives another error, there's an installation problem that
+you should fix before trying to go any further.  If you get a nicely
+formatted listing of the environment and form content (in this
+example, the fields should be listed as ``addr'' with value ``At Home''
+and ``name'' with value ``Joe Blow''), the \file{cgi.py} script has been
+installed correctly.  If you follow the same procedure for your own
+script, you should now be able to debug it.
+
+The next step could be to call the \module{cgi} module's
+\function{test()} function from your script: replace its main code
+with the single statement
+
+\begin{verbatim}
+cgi.test()
+\end{verbatim}
+
+This should produce the same results as those gotten from installing
+the \file{cgi.py} file itself.
+
+When an ordinary Python script raises an unhandled exception (for
+whatever reason: of a typo in a module name, a file that can't be
+opened, etc.), the Python interpreter prints a nice traceback and
+exits.  While the Python interpreter will still do this when your CGI
+script raises an exception, most likely the traceback will end up in
+one of the HTTP server's log files, or be discarded altogether.
+
+Fortunately, once you have managed to get your script to execute
+\emph{some} code, you can easily send tracebacks to the Web browser
+using the \refmodule{cgitb} module.  If you haven't done so already,
+just add the line:
+
+\begin{verbatim}
+import cgitb; cgitb.enable()
+\end{verbatim}
+
+to the top of your script.  Then try running it again; when a
+problem occurs, you should see a detailed report that will
+likely make apparent the cause of the crash.
+
+If you suspect that there may be a problem in importing the
+\refmodule{cgitb} module, you can use an even more robust approach
+(which only uses built-in modules):
+
+\begin{verbatim}
+import sys
+sys.stderr = sys.stdout
+print "Content-Type: text/plain"
+print
+...your code here...
+\end{verbatim}
+
+This relies on the Python interpreter to print the traceback.  The
+content type of the output is set to plain text, which disables all
+HTML processing.  If your script works, the raw HTML will be displayed
+by your client.  If it raises an exception, most likely after the
+first two lines have been printed, a traceback will be displayed.
+Because no HTML interpretation is going on, the traceback will be
+readable.
+
+
+\subsection{Common problems and solutions}
+
+\begin{itemize}
+\item Most HTTP servers buffer the output from CGI scripts until the
+script is completed.  This means that it is not possible to display a
+progress report on the client's display while the script is running.
+
+\item Check the installation instructions above.
+
+\item Check the HTTP server's log files.  (\samp{tail -f logfile} in a
+separate window may be useful!)
+
+\item Always check a script for syntax errors first, by doing something
+like \samp{python script.py}.
+
+\item If your script does not have any syntax errors, try adding
+\samp{import cgitb; cgitb.enable()} to the top of the script.
+
+\item When invoking external programs, make sure they can be found.
+Usually, this means using absolute path names --- \envvar{PATH} is
+usually not set to a very useful value in a CGI script.
+
+\item When reading or writing external files, make sure they can be read
+or written by the userid under which your CGI script will be running:
+this is typically the userid under which the web server is running, or some
+explicitly specified userid for a web server's \samp{suexec} feature.
+
+\item Don't try to give a CGI script a set-uid mode.  This doesn't work on
+most systems, and is a security liability as well.
+\end{itemize}
+