diff options
author | cinap_lenrek <cinap_lenrek@localhost> | 2011-05-03 11:25:13 +0000 |
---|---|---|
committer | cinap_lenrek <cinap_lenrek@localhost> | 2011-05-03 11:25:13 +0000 |
commit | 458120dd40db6b4df55a4e96b650e16798ef06a0 (patch) | |
tree | 8f82685be24fef97e715c6f5ca4c68d34d5074ee /sys/src/cmd/python/Doc/tut | |
parent | 3a742c699f6806c1145aea5149bf15de15a0afd7 (diff) |
add hg and python
Diffstat (limited to 'sys/src/cmd/python/Doc/tut')
-rw-r--r-- | sys/src/cmd/python/Doc/tut/glossary.tex | 352 | ||||
-rw-r--r-- | sys/src/cmd/python/Doc/tut/tut.tex | 5907 |
2 files changed, 6259 insertions, 0 deletions
diff --git a/sys/src/cmd/python/Doc/tut/glossary.tex b/sys/src/cmd/python/Doc/tut/glossary.tex new file mode 100644 index 000000000..17cc76776 --- /dev/null +++ b/sys/src/cmd/python/Doc/tut/glossary.tex @@ -0,0 +1,352 @@ +\chapter{Glossary\label{glossary}} + +%%% keep the entries sorted and include at least one \index{} item for each +%%% cross-references are marked with \emph{entry} + +\begin{description} + + +\index{>>>} +\item[\code{>>>}] +The typical Python prompt of the interactive shell. Often seen for +code examples that can be tried right away in the interpreter. + +\index{...} +\item[\code{.\code{.}.}] +The typical Python prompt of the interactive shell when entering code +for an indented code block. + +\index{BDFL} +\item[BDFL] +Benevolent Dictator For Life, a.k.a. \ulink{Guido van +Rossum}{http://www.python.org/\textasciitilde{}guido/}, Python's creator. + +\index{byte code} +\item[byte code] +The internal representation of a Python program in the interpreter. +The byte code is also cached in \code{.pyc} and \code{.pyo} +files so that executing the same file is faster the second time +(recompilation from source to byte code can be avoided). This +``intermediate language'' is said to run on a ``virtual +machine'' that calls the subroutines corresponding to each bytecode. + +\index{classic class} +\item[classic class] +Any class which does not inherit from \class{object}. See +\emph{new-style class}. + +\index{coercion} +\item[coercion] +The implicit conversion of an instance of one type to another during an +operation which involves two arguments of the same type. For example, +{}\code{int(3.15)} converts the floating point number to the integer +{}\code{3}, but in {}\code{3+4.5}, each argument is of a different type (one +int, one float), and both must be converted to the same type before they can +be added or it will raise a {}\code{TypeError}. Coercion between two +operands can be performed with the {}\code{coerce} builtin function; thus, +{}\code{3+4.5} is equivalent to calling {}\code{operator.add(*coerce(3, +4.5))} and results in {}\code{operator.add(3.0, 4.5)}. Without coercion, +all arguments of even compatible types would have to be normalized to the +same value by the programmer, e.g., {}\code{float(3)+4.5} rather than just +{}\code{3+4.5}. + +\index{complex number} +\item[complex number] +An extension of the familiar real number system in which all numbers are +expressed as a sum of a real part and an imaginary part. Imaginary numbers +are real multiples of the imaginary unit (the square root of {}\code{-1}), +often written {}\code{i} in mathematics or {}\code{j} in engineering. +Python has builtin support for complex numbers, which are written with this +latter notation; the imaginary part is written with a {}\code{j} suffix, +e.g., {}\code{3+1j}. To get access to complex equivalents of the +{}\module{math} module, use {}\module{cmath}. Use of complex numbers is a +fairly advanced mathematical feature. If you're not aware of a need for them, +it's almost certain you can safely ignore them. + +\index{descriptor} +\item[descriptor] +Any \emph{new-style} object that defines the methods +{}\method{__get__()}, \method{__set__()}, or \method{__delete__()}. +When a class attribute is a descriptor, its special binding behavior +is triggered upon attribute lookup. Normally, writing \var{a.b} looks +up the object \var{b} in the class dictionary for \var{a}, but if +{}\var{b} is a descriptor, the defined method gets called. +Understanding descriptors is a key to a deep understanding of Python +because they are the basis for many features including functions, +methods, properties, class methods, static methods, and reference to +super classes. + +\index{dictionary} +\item[dictionary] +An associative array, where arbitrary keys are mapped to values. The +use of \class{dict} much resembles that for \class{list}, but the keys +can be any object with a \method{__hash__()} function, not just +integers starting from zero. Called a hash in Perl. + +\index{duck-typing} +\item[duck-typing] +Pythonic programming style that determines an object's type by inspection +of its method or attribute signature rather than by explicit relationship +to some type object ("If it looks like a duck and quacks like a duck, it +must be a duck.") By emphasizing interfaces rather than specific types, +well-designed code improves its flexibility by allowing polymorphic +substitution. Duck-typing avoids tests using \function{type()} or +\function{isinstance()}. Instead, it typically employs +\function{hasattr()} tests or {}\emph{EAFP} programming. + +\index{EAFP} +\item[EAFP] +Easier to ask for forgiveness than permission. This common Python +coding style assumes the existence of valid keys or attributes and +catches exceptions if the assumption proves false. This clean and +fast style is characterized by the presence of many \keyword{try} and +{}\keyword{except} statements. The technique contrasts with the +{}\emph{LBYL} style that is common in many other languages such as C. + +\index{__future__} +\item[__future__] +A pseudo module which programmers can use to enable new language +features which are not compatible with the current interpreter. For +example, the expression \code{11/4} currently evaluates to \code{2}. +If the module in which it is executed had enabled \emph{true division} +by executing: + +\begin{verbatim} +from __future__ import division +\end{verbatim} + +the expression \code{11/4} would evaluate to \code{2.75}. By +importing the \ulink{\module{__future__}}{../lib/module-future.html} +module and evaluating its variables, you can see when a new feature +was first added to the language and when it will become the default: + +\begin{verbatim} +>>> import __future__ +>>> __future__.division +_Feature((2, 2, 0, 'alpha', 2), (3, 0, 0, 'alpha', 0), 8192) +\end{verbatim} + +\index{generator} +\item[generator] +A function that returns an iterator. It looks like a normal function except +that values are returned to the caller using a \keyword{yield} statement +instead of a {}\keyword{return} statement. Generator functions often +contain one or more {}\keyword{for} or \keyword{while} loops that +\keyword{yield} elements back to the caller. The function execution is +stopped at the {}\keyword{yield} keyword (returning the result) and is +resumed there when the next element is requested by calling the +\method{next()} method of the returned iterator. + +\index{generator expression} +\item[generator expression] +An expression that returns a generator. It looks like a normal expression +followed by a \keyword{for} expression defining a loop variable, range, and +an optional \keyword{if} expression. The combined expression generates +values for an enclosing function: + +\begin{verbatim} +>>> sum(i*i for i in range(10)) # sum of squares 0, 1, 4, ... 81 +285 +\end{verbatim} + +\index{GIL} +\item[GIL] +See \emph{global interpreter lock}. + +\index{global interpreter lock} +\item[global interpreter lock] +The lock used by Python threads to assure that only one thread can be +run at a time. This simplifies Python by assuring that no two +processes can access the same memory at the same time. Locking the +entire interpreter makes it easier for the interpreter to be +multi-threaded, at the expense of some parallelism on multi-processor +machines. Efforts have been made in the past to create a +``free-threaded'' interpreter (one which locks shared data at a much +finer granularity), but performance suffered in the common +single-processor case. + +\index{IDLE} +\item[IDLE] +An Integrated Development Environment for Python. IDLE is a +basic editor and interpreter environment that ships with the standard +distribution of Python. Good for beginners, it also serves as clear +example code for those wanting to implement a moderately +sophisticated, multi-platform GUI application. + +\index{immutable} +\item[immutable] +An object with fixed value. Immutable objects are numbers, strings or +tuples (and more). Such an object cannot be altered. A new object +has to be created if a different value has to be stored. They play an +important role in places where a constant hash value is needed, for +example as a key in a dictionary. + +\index{integer division} +\item[integer division] +Mathematical division discarding any remainder. For example, the +expression \code{11/4} currently evaluates to \code{2} in contrast +to the \code{2.75} returned by float division. Also called +{}\emph{floor division}. When dividing two integers the outcome will +always be another integer (having the floor function applied to it). +However, if one of the operands is another numeric type (such as a +{}\class{float}), the result will be coerced (see \emph{coercion}) to +a common type. For example, an integer divided by a float will result +in a float value, possibly with a decimal fraction. Integer division +can be forced by using the \code{//} operator instead of the \code{/} +operator. See also \emph{__future__}. + +\index{interactive} +\item[interactive] +Python has an interactive interpreter which means that you can try out +things and immediately see their results. Just launch \code{python} with no +arguments (possibly by selecting it from your computer's main menu). +It is a very powerful way to test out new ideas or inspect modules and +packages (remember \code{help(x)}). + +\index{interpreted} +\item[interpreted] +Python is an interpreted language, as opposed to a compiled one. This means +that the source files can be run directly without first creating an +executable which is then run. Interpreted languages typically have a +shorter development/debug cycle than compiled ones, though their programs +generally also run more slowly. See also {}\emph{interactive}. + +\index{iterable} +\item[iterable] +A container object capable of returning its members one at a time. +Examples of iterables include all sequence types (such as \class{list}, +{}\class{str}, and \class{tuple}) and some non-sequence types like +{}\class{dict} and \class{file} and objects of any classes you define +with an \method{__iter__()} or \method{__getitem__()} method. Iterables +can be used in a \keyword{for} loop and in many other places where a +sequence is needed (\function{zip()}, \function{map()}, ...). When an +iterable object is passed as an argument to the builtin function +{}\function{iter()}, it returns an iterator for the object. This +iterator is good for one pass over the set of values. When using +iterables, it is usually not necessary to call \function{iter()} or +deal with iterator objects yourself. The \code{for} statement does +that automatically for you, creating a temporary unnamed variable to +hold the iterator for the duration of the loop. See also +{}\emph{iterator}, \emph{sequence}, and \emph{generator}. + +\index{iterator} +\item[iterator] +An object representing a stream of data. Repeated calls to the +iterator's \method{next()} method return successive items in the +stream. When no more data is available a \exception{StopIteration} +exception is raised instead. At this point, the iterator object is +exhausted and any further calls to its \method{next()} method just +raise \exception{StopIteration} again. Iterators are required to have +an \method{__iter__()} method that returns the iterator object +itself so every iterator is also iterable and may be used in most +places where other iterables are accepted. One notable exception is +code that attempts multiple iteration passes. A container object +(such as a \class{list}) produces a fresh new iterator each time you +pass it to the \function{iter()} function or use it in a +{}\keyword{for} loop. Attempting this with an iterator will just +return the same exhausted iterator object used in the previous iteration +pass, making it appear like an empty container. + +\index{LBYL} +\item[LBYL] +Look before you leap. This coding style explicitly tests for +pre-conditions before making calls or lookups. This style contrasts +with the \emph{EAFP} approach and is characterized by the presence of +many \keyword{if} statements. + +\index{list comprehension} +\item[list comprehension] +A compact way to process all or a subset of elements in a sequence and +return a list with the results. \code{result = ["0x\%02x" +\% x for x in range(256) if x \% 2 == 0]} generates a list of strings +containing hex numbers (0x..) that are even and in the range from 0 to 255. +The \keyword{if} clause is optional. If omitted, all elements in +{}\code{range(256)} are processed. + +\index{mapping} +\item[mapping] +A container object (such as \class{dict}) that supports arbitrary key +lookups using the special method \method{__getitem__()}. + +\index{metaclass} +\item[metaclass] +The class of a class. Class definitions create a class name, a class +dictionary, and a list of base classes. The metaclass is responsible +for taking those three arguments and creating the class. Most object +oriented programming languages provide a default implementation. What +makes Python special is that it is possible to create custom +metaclasses. Most users never need this tool, but when the need +arises, metaclasses can provide powerful, elegant solutions. They +have been used for logging attribute access, adding thread-safety, +tracking object creation, implementing singletons, and many other +tasks. + +\index{mutable} +\item[mutable] +Mutable objects can change their value but keep their \function{id()}. +See also \emph{immutable}. + +\index{namespace} +\item[namespace] +The place where a variable is stored. Namespaces are implemented as +dictionaries. There are the local, global and builtin namespaces +as well as nested namespaces in objects (in methods). Namespaces support +modularity by preventing naming conflicts. For instance, the +functions \function{__builtin__.open()} and \function{os.open()} are +distinguished by their namespaces. Namespaces also aid readability +and maintainability by making it clear which module implements a +function. For instance, writing \function{random.seed()} or +{}\function{itertools.izip()} makes it clear that those functions are +implemented by the \ulink{\module{random}}{../lib/module-random.html} +and \ulink{\module{itertools}}{../lib/module-itertools.html} modules +respectively. + +\index{nested scope} +\item[nested scope] +The ability to refer to a variable in an enclosing definition. For +instance, a function defined inside another function can refer to +variables in the outer function. Note that nested scopes work only +for reference and not for assignment which will always write to the +innermost scope. In contrast, local variables both read and write in +the innermost scope. Likewise, global variables read and write to the +global namespace. + +\index{new-style class} +\item[new-style class] +Any class that inherits from \class{object}. This includes all +built-in types like \class{list} and \class{dict}. Only new-style +classes can use Python's newer, versatile features like +{}\method{__slots__}, descriptors, properties, +\method{__getattribute__()}, class methods, and static methods. + +\index{Python3000} +\item[Python3000] +A mythical python release, not required to be backward compatible, with +telepathic interface. + +\index{__slots__} +\item[__slots__] +A declaration inside a \emph{new-style class} that saves memory by +pre-declaring space for instance attributes and eliminating instance +dictionaries. Though popular, the technique is somewhat tricky to get +right and is best reserved for rare cases where there are large +numbers of instances in a memory-critical application. + +\index{sequence} +\item[sequence] +An \emph{iterable} which supports efficient element access using +integer indices via the \method{__getitem__()} and +{}\method{__len__()} special methods. Some built-in sequence types +are \class{list}, \class{str}, \class{tuple}, and \class{unicode}. +Note that \class{dict} also supports \method{__getitem__()} and +{}\method{__len__()}, but is considered a mapping rather than a +sequence because the lookups use arbitrary \emph{immutable} keys +rather than integers. + +\index{Zen of Python} +\item[Zen of Python] +Listing of Python design principles and philosophies that are helpful +in understanding and using the language. The listing can be found by +typing ``\code{import this}'' at the interactive prompt. + +\end{description} diff --git a/sys/src/cmd/python/Doc/tut/tut.tex b/sys/src/cmd/python/Doc/tut/tut.tex new file mode 100644 index 000000000..2981a789d --- /dev/null +++ b/sys/src/cmd/python/Doc/tut/tut.tex @@ -0,0 +1,5907 @@ +\documentclass{manual} +\usepackage[T1]{fontenc} +\usepackage{textcomp} + +% Things to do: +% Should really move the Python startup file info to an appendix + +\title{Python Tutorial} + +\input{boilerplate} + +\makeindex + +\begin{document} + +\maketitle + +\ifhtml +\chapter*{Front Matter\label{front}} +\fi + +\input{copyright} + +\begin{abstract} + +\noindent +Python is an easy to learn, powerful programming language. It has +efficient high-level data structures and a simple but effective +approach to object-oriented programming. Python's elegant syntax and +dynamic typing, together with its interpreted nature, make it an ideal +language for scripting and rapid application development in many areas +on most platforms. + +The Python interpreter and the extensive standard library are freely +available in source or binary form for all major platforms from the +Python Web site, \url{http://www.python.org/}, and may be freely +distributed. The same site also contains distributions of and +pointers to many free third party Python modules, programs and tools, +and additional documentation. + +The Python interpreter is easily extended with new functions and data +types implemented in C or \Cpp{} (or other languages callable from C). +Python is also suitable as an extension language for customizable +applications. + +This tutorial introduces the reader informally to the basic concepts +and features of the Python language and system. It helps to have a +Python interpreter handy for hands-on experience, but all examples are +self-contained, so the tutorial can be read off-line as well. + +For a description of standard objects and modules, see the +\citetitle[../lib/lib.html]{Python Library Reference} document. The +\citetitle[../ref/ref.html]{Python Reference Manual} gives a more +formal definition of the language. To write extensions in C or +\Cpp, read \citetitle[../ext/ext.html]{Extending and Embedding the +Python Interpreter} and \citetitle[../api/api.html]{Python/C API +Reference}. There are also several books covering Python in depth. + +This tutorial does not attempt to be comprehensive and cover every +single feature, or even every commonly used feature. Instead, it +introduces many of Python's most noteworthy features, and will give +you a good idea of the language's flavor and style. After reading it, +you will be able to read and write Python modules and programs, and +you will be ready to learn more about the various Python library +modules described in the \citetitle[../lib/lib.html]{Python Library +Reference}. + +\end{abstract} + +\tableofcontents + + +\chapter{Whetting Your Appetite \label{intro}} + +If you do much work on computers, eventually you find that there's +some task you'd like to automate. For example, you may wish to +perform a search-and-replace over a large number of text files, or +rename and rearrange a bunch of photo files in a complicated way. +Perhaps you'd like to write a small custom database, or a specialized +GUI application, or a simple game. + +If you're a professional software developer, you may have to work with +several C/\Cpp/Java libraries but find the usual +write/compile/test/re-compile cycle is too slow. Perhaps you're +writing a test suite for such a library and find writing the testing +code a tedious task. Or maybe you've written a program that could use +an extension language, and you don't want to design and implement a +whole new language for your application. + +Python is just the language for you. + +You could write a {\UNIX} shell script or Windows batch files for some +of these tasks, but shell scripts are best at moving around files and +changing text data, not well-suited for GUI applications or games. +You could write a C/{\Cpp}/Java program, but it can take a lot of +development time to get even a first-draft program. Python is simpler +to use, available on Windows, MacOS X, and {\UNIX} operating systems, +and will help you get the job done more quickly. + +Python is simple to use, but it is a real programming language, +offering much more structure and support for large programs than shell +scripts or batch files can offer. On the other hand, Python also +offers much more error checking than C, and, being a +\emph{very-high-level language}, it has high-level data types built +in, such as flexible arrays and dictionaries. Because of its more +general data types Python is applicable to a much larger problem +domain than Awk or even Perl, yet many things are at +least as easy in Python as in those languages. + +Python allows you to split your program into modules that can be +reused in other Python programs. It comes with a large collection of +standard modules that you can use as the basis of your programs --- or +as examples to start learning to program in Python. Some of these +modules provide things like file I/O, system calls, +sockets, and even interfaces to graphical user interface toolkits like Tk. + +Python is an interpreted language, which can save you considerable time +during program development because no compilation and linking is +necessary. The interpreter can be used interactively, which makes it +easy to experiment with features of the language, to write throw-away +programs, or to test functions during bottom-up program development. +It is also a handy desk calculator. + +Python enables programs to be written compactly and readably. Programs +written in Python are typically much shorter than equivalent C, +\Cpp{}, or Java programs, for several reasons: +\begin{itemize} +\item +the high-level data types allow you to express complex operations in a +single statement; +\item +statement grouping is done by indentation instead of beginning and ending +brackets; +\item +no variable or argument declarations are necessary. +\end{itemize} + +Python is \emph{extensible}: if you know how to program in C it is easy +to add a new built-in function or module to the interpreter, either to +perform critical operations at maximum speed, or to link Python +programs to libraries that may only be available in binary form (such +as a vendor-specific graphics library). Once you are really hooked, +you can link the Python interpreter into an application written in C +and use it as an extension or command language for that application. + +By the way, the language is named after the BBC show ``Monty Python's +Flying Circus'' and has nothing to do with nasty reptiles. Making +references to Monty Python skits in documentation is not only allowed, +it is encouraged! + +%\section{Where From Here \label{where}} + +Now that you are all excited about Python, you'll want to examine it +in some more detail. Since the best way to learn a language is +to use it, the tutorial invites you to play with the Python interpreter +as you read. + +In the next chapter, the mechanics of using the interpreter are +explained. This is rather mundane information, but essential for +trying out the examples shown later. + +The rest of the tutorial introduces various features of the Python +language and system through examples, beginning with simple +expressions, statements and data types, through functions and modules, +and finally touching upon advanced concepts like exceptions +and user-defined classes. + +\chapter{Using the Python Interpreter \label{using}} + +\section{Invoking the Interpreter \label{invoking}} + +The Python interpreter is usually installed as +\file{/usr/local/bin/python} on those machines where it is available; +putting \file{/usr/local/bin} in your \UNIX{} shell's search path +makes it possible to start it by typing the command + +\begin{verbatim} +python +\end{verbatim} + +to the shell. Since the choice of the directory where the interpreter +lives is an installation option, other places are possible; check with +your local Python guru or system administrator. (E.g., +\file{/usr/local/python} is a popular alternative location.) + +On Windows machines, the Python installation is usually placed in +\file{C:\e Python24}, though you can change this when you're running +the installer. To add this directory to your path, +you can type the following command into the command prompt in a DOS box: + +\begin{verbatim} +set path=%path%;C:\python24 +\end{verbatim} + + +Typing an end-of-file character (\kbd{Control-D} on \UNIX, +\kbd{Control-Z} on Windows) at the primary prompt causes the +interpreter to exit with a zero exit status. If that doesn't work, +you can exit the interpreter by typing the following commands: +\samp{import sys; sys.exit()}. + +The interpreter's line-editing features usually aren't very +sophisticated. On \UNIX, whoever installed the interpreter may have +enabled support for the GNU readline library, which adds more +elaborate interactive editing and history features. Perhaps the +quickest check to see whether command line editing is supported is +typing Control-P to the first Python prompt you get. If it beeps, you +have command line editing; see Appendix \ref{interacting} for an +introduction to the keys. If nothing appears to happen, or if +\code{\^P} is echoed, command line editing isn't available; you'll +only be able to use backspace to remove characters from the current +line. + +The interpreter operates somewhat like the \UNIX{} shell: when called +with standard input connected to a tty device, it reads and executes +commands interactively; when called with a file name argument or with +a file as standard input, it reads and executes a \emph{script} from +that file. + +A second way of starting the interpreter is +\samp{\program{python} \programopt{-c} \var{command} [arg] ...}, which +executes the statement(s) in \var{command}, analogous to the shell's +\programopt{-c} option. Since Python statements often contain spaces +or other characters that are special to the shell, it is best to quote +\var{command} in its entirety with double quotes. + +Some Python modules are also useful as scripts. These can be invoked using +\samp{\program{python} \programopt{-m} \var{module} [arg] ...}, which +executes the source file for \var{module} as if you had spelled out its +full name on the command line. + +Note that there is a difference between \samp{python file} and +\samp{python <file}. In the latter case, input requests from the +program, such as calls to \function{input()} and \function{raw_input()}, are +satisfied from \emph{file}. Since this file has already been read +until the end by the parser before the program starts executing, the +program will encounter end-of-file immediately. In the former case +(which is usually what you want) they are satisfied from whatever file +or device is connected to standard input of the Python interpreter. + +When a script file is used, it is sometimes useful to be able to run +the script and enter interactive mode afterwards. This can be done by +passing \programopt{-i} before the script. (This does not work if the +script is read from standard input, for the same reason as explained +in the previous paragraph.) + +\subsection{Argument Passing \label{argPassing}} + +When known to the interpreter, the script name and additional +arguments thereafter are passed to the script in the variable +\code{sys.argv}, which is a list of strings. Its length is at least +one; when no script and no arguments are given, \code{sys.argv[0]} is +an empty string. When the script name is given as \code{'-'} (meaning +standard input), \code{sys.argv[0]} is set to \code{'-'}. When +\programopt{-c} \var{command} is used, \code{sys.argv[0]} is set to +\code{'-c'}. When \programopt{-m} \var{module} is used, \code{sys.argv[0]} +is set to the full name of the located module. Options found after +\programopt{-c} \var{command} or \programopt{-m} \var{module} are not consumed +by the Python interpreter's option processing but left in \code{sys.argv} for +the command or module to handle. + +\subsection{Interactive Mode \label{interactive}} + +When commands are read from a tty, the interpreter is said to be in +\emph{interactive mode}. In this mode it prompts for the next command +with the \emph{primary prompt}, usually three greater-than signs +(\samp{>>>~}); for continuation lines it prompts with the +\emph{secondary prompt}, by default three dots (\samp{...~}). +The interpreter prints a welcome message stating its version number +and a copyright notice before printing the first prompt: + +\begin{verbatim} +python +Python 1.5.2b2 (#1, Feb 28 1999, 00:02:06) [GCC 2.8.1] on sunos5 +Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam +>>> +\end{verbatim} + +Continuation lines are needed when entering a multi-line construct. +As an example, take a look at this \keyword{if} statement: + +\begin{verbatim} +>>> the_world_is_flat = 1 +>>> if the_world_is_flat: +... print "Be careful not to fall off!" +... +Be careful not to fall off! +\end{verbatim} + + +\section{The Interpreter and Its Environment \label{interp}} + +\subsection{Error Handling \label{error}} + +When an error occurs, the interpreter prints an error +message and a stack trace. In interactive mode, it then returns to +the primary prompt; when input came from a file, it exits with a +nonzero exit status after printing +the stack trace. (Exceptions handled by an \keyword{except} clause in a +\keyword{try} statement are not errors in this context.) Some errors are +unconditionally fatal and cause an exit with a nonzero exit; this +applies to internal inconsistencies and some cases of running out of +memory. All error messages are written to the standard error stream; +normal output from executed commands is written to standard +output. + +Typing the interrupt character (usually Control-C or DEL) to the +primary or secondary prompt cancels the input and returns to the +primary prompt.\footnote{ + A problem with the GNU Readline package may prevent this. +} +Typing an interrupt while a command is executing raises the +\exception{KeyboardInterrupt} exception, which may be handled by a +\keyword{try} statement. + +\subsection{Executable Python Scripts \label{scripts}} + +On BSD'ish \UNIX{} systems, Python scripts can be made directly +executable, like shell scripts, by putting the line + +\begin{verbatim} +#! /usr/bin/env python +\end{verbatim} + +(assuming that the interpreter is on the user's \envvar{PATH}) at the +beginning of the script and giving the file an executable mode. The +\samp{\#!} must be the first two characters of the file. On some +platforms, this first line must end with a \UNIX-style line ending +(\character{\e n}), not a Mac OS (\character{\e r}) or Windows +(\character{\e r\e n}) line ending. Note that +the hash, or pound, character, \character{\#}, is used to start a +comment in Python. + +The script can be given an executable mode, or permission, using the +\program{chmod} command: + +\begin{verbatim} +$ chmod +x myscript.py +\end{verbatim} % $ <-- bow to font-lock + + +\subsection{Source Code Encoding} + +It is possible to use encodings different than \ASCII{} in Python source +files. The best way to do it is to put one more special comment line +right after the \code{\#!} line to define the source file encoding: + +\begin{alltt} +# -*- coding: \var{encoding} -*- +\end{alltt} + +With that declaration, all characters in the source file will be treated as +having the encoding \var{encoding}, and it will be +possible to directly write Unicode string literals in the selected +encoding. The list of possible encodings can be found in the +\citetitle[../lib/lib.html]{Python Library Reference}, in the section +on \ulink{\module{codecs}}{../lib/module-codecs.html}. + +For example, to write Unicode literals including the Euro currency +symbol, the ISO-8859-15 encoding can be used, with the Euro symbol +having the ordinal value 164. This script will print the value 8364 +(the Unicode codepoint corresponding to the Euro symbol) and then +exit: + +\begin{alltt} +# -*- coding: iso-8859-15 -*- + +currency = u"\texteuro" +print ord(currency) +\end{alltt} + +If your editor supports saving files as \code{UTF-8} with a UTF-8 +\emph{byte order mark} (aka BOM), you can use that instead of an +encoding declaration. IDLE supports this capability if +\code{Options/General/Default Source Encoding/UTF-8} is set. Notice +that this signature is not understood in older Python releases (2.2 +and earlier), and also not understood by the operating system for +script files with \code{\#!} lines (only used on \UNIX{} systems). + +By using UTF-8 (either through the signature or an encoding +declaration), characters of most languages in the world can be used +simultaneously in string literals and comments. Using non-\ASCII{} +characters in identifiers is not supported. To display all these +characters properly, your editor must recognize that the file is +UTF-8, and it must use a font that supports all the characters in the +file. + +\subsection{The Interactive Startup File \label{startup}} + +% XXX This should probably be dumped in an appendix, since most people +% don't use Python interactively in non-trivial ways. + +When you use Python interactively, it is frequently handy to have some +standard commands executed every time the interpreter is started. You +can do this by setting an environment variable named +\envvar{PYTHONSTARTUP} to the name of a file containing your start-up +commands. This is similar to the \file{.profile} feature of the +\UNIX{} shells. + +This file is only read in interactive sessions, not when Python reads +commands from a script, and not when \file{/dev/tty} is given as the +explicit source of commands (which otherwise behaves like an +interactive session). It is executed in the same namespace where +interactive commands are executed, so that objects that it defines or +imports can be used without qualification in the interactive session. +You can also change the prompts \code{sys.ps1} and \code{sys.ps2} in +this file. + +If you want to read an additional start-up file from the current +directory, you can program this in the global start-up file using code +like \samp{if os.path.isfile('.pythonrc.py'): +execfile('.pythonrc.py')}. If you want to use the startup file in a +script, you must do this explicitly in the script: + +\begin{verbatim} +import os +filename = os.environ.get('PYTHONSTARTUP') +if filename and os.path.isfile(filename): + execfile(filename) +\end{verbatim} + + +\chapter{An Informal Introduction to Python \label{informal}} + +In the following examples, input and output are distinguished by the +presence or absence of prompts (\samp{>>>~} and \samp{...~}): to repeat +the example, you must type everything after the prompt, when the +prompt appears; lines that do not begin with a prompt are output from +the interpreter. % +%\footnote{ +% I'd prefer to use different fonts to distinguish input +% from output, but the amount of LaTeX hacking that would require +% is currently beyond my ability. +%} +Note that a secondary prompt on a line by itself in an example means +you must type a blank line; this is used to end a multi-line command. + +Many of the examples in this manual, even those entered at the +interactive prompt, include comments. Comments in Python start with +the hash character, \character{\#}, and extend to the end of the +physical line. A comment may appear at the start of a line or +following whitespace or code, but not within a string literal. A hash +character within a string literal is just a hash character. + +Some examples: + +\begin{verbatim} +# this is the first comment +SPAM = 1 # and this is the second comment + # ... and now a third! +STRING = "# This is not a comment." +\end{verbatim} + + +\section{Using Python as a Calculator \label{calculator}} + +Let's try some simple Python commands. Start the interpreter and wait +for the primary prompt, \samp{>>>~}. (It shouldn't take long.) + +\subsection{Numbers \label{numbers}} + +The interpreter acts as a simple calculator: you can type an +expression at it and it will write the value. Expression syntax is +straightforward: the operators \code{+}, \code{-}, \code{*} and +\code{/} work just like in most other languages (for example, Pascal +or C); parentheses can be used for grouping. For example: + +\begin{verbatim} +>>> 2+2 +4 +>>> # This is a comment +... 2+2 +4 +>>> 2+2 # and a comment on the same line as code +4 +>>> (50-5*6)/4 +5 +>>> # Integer division returns the floor: +... 7/3 +2 +>>> 7/-3 +-3 +\end{verbatim} + +The equal sign (\character{=}) is used to assign a value to a variable. +Afterwards, no result is displayed before the next interactive prompt: + +\begin{verbatim} +>>> width = 20 +>>> height = 5*9 +>>> width * height +900 +\end{verbatim} + +A value can be assigned to several variables simultaneously: + +\begin{verbatim} +>>> x = y = z = 0 # Zero x, y and z +>>> x +0 +>>> y +0 +>>> z +0 +\end{verbatim} + +There is full support for floating point; operators with mixed type +operands convert the integer operand to floating point: + +\begin{verbatim} +>>> 3 * 3.75 / 1.5 +7.5 +>>> 7.0 / 2 +3.5 +\end{verbatim} + +Complex numbers are also supported; imaginary numbers are written with +a suffix of \samp{j} or \samp{J}. Complex numbers with a nonzero +real component are written as \samp{(\var{real}+\var{imag}j)}, or can +be created with the \samp{complex(\var{real}, \var{imag})} function. + +\begin{verbatim} +>>> 1j * 1J +(-1+0j) +>>> 1j * complex(0,1) +(-1+0j) +>>> 3+1j*3 +(3+3j) +>>> (3+1j)*3 +(9+3j) +>>> (1+2j)/(1+1j) +(1.5+0.5j) +\end{verbatim} + +Complex numbers are always represented as two floating point numbers, +the real and imaginary part. To extract these parts from a complex +number \var{z}, use \code{\var{z}.real} and \code{\var{z}.imag}. + +\begin{verbatim} +>>> a=1.5+0.5j +>>> a.real +1.5 +>>> a.imag +0.5 +\end{verbatim} + +The conversion functions to floating point and integer +(\function{float()}, \function{int()} and \function{long()}) don't +work for complex numbers --- there is no one correct way to convert a +complex number to a real number. Use \code{abs(\var{z})} to get its +magnitude (as a float) or \code{z.real} to get its real part. + +\begin{verbatim} +>>> a=3.0+4.0j +>>> float(a) +Traceback (most recent call last): + File "<stdin>", line 1, in ? +TypeError: can't convert complex to float; use abs(z) +>>> a.real +3.0 +>>> a.imag +4.0 +>>> abs(a) # sqrt(a.real**2 + a.imag**2) +5.0 +>>> +\end{verbatim} + +In interactive mode, the last printed expression is assigned to the +variable \code{_}. This means that when you are using Python as a +desk calculator, it is somewhat easier to continue calculations, for +example: + +\begin{verbatim} +>>> tax = 12.5 / 100 +>>> price = 100.50 +>>> price * tax +12.5625 +>>> price + _ +113.0625 +>>> round(_, 2) +113.06 +>>> +\end{verbatim} + +This variable should be treated as read-only by the user. Don't +explicitly assign a value to it --- you would create an independent +local variable with the same name masking the built-in variable with +its magic behavior. + +\subsection{Strings \label{strings}} + +Besides numbers, Python can also manipulate strings, which can be +expressed in several ways. They can be enclosed in single quotes or +double quotes: + +\begin{verbatim} +>>> 'spam eggs' +'spam eggs' +>>> 'doesn\'t' +"doesn't" +>>> "doesn't" +"doesn't" +>>> '"Yes," he said.' +'"Yes," he said.' +>>> "\"Yes,\" he said." +'"Yes," he said.' +>>> '"Isn\'t," she said.' +'"Isn\'t," she said.' +\end{verbatim} + +String literals can span multiple lines in several ways. Continuation +lines can be used, with a backslash as the last character on the line +indicating that the next line is a logical continuation of the line: + +\begin{verbatim} +hello = "This is a rather long string containing\n\ +several lines of text just as you would do in C.\n\ + Note that whitespace at the beginning of the line is\ + significant." + +print hello +\end{verbatim} + +Note that newlines still need to be embedded in the string using +\code{\e n}; the newline following the trailing backslash is +discarded. This example would print the following: + +\begin{verbatim} +This is a rather long string containing +several lines of text just as you would do in C. + Note that whitespace at the beginning of the line is significant. +\end{verbatim} + +If we make the string literal a ``raw'' string, however, the +\code{\e n} sequences are not converted to newlines, but the backslash +at the end of the line, and the newline character in the source, are +both included in the string as data. Thus, the example: + +\begin{verbatim} +hello = r"This is a rather long string containing\n\ +several lines of text much as you would do in C." + +print hello +\end{verbatim} + +would print: + +\begin{verbatim} +This is a rather long string containing\n\ +several lines of text much as you would do in C. +\end{verbatim} + +Or, strings can be surrounded in a pair of matching triple-quotes: +\code{"""} or \code{'\code{'}'}. End of lines do not need to be escaped +when using triple-quotes, but they will be included in the string. + +\begin{verbatim} +print """ +Usage: thingy [OPTIONS] + -h Display this usage message + -H hostname Hostname to connect to +""" +\end{verbatim} + +produces the following output: + +\begin{verbatim} +Usage: thingy [OPTIONS] + -h Display this usage message + -H hostname Hostname to connect to +\end{verbatim} + +The interpreter prints the result of string operations in the same way +as they are typed for input: inside quotes, and with quotes and other +funny characters escaped by backslashes, to show the precise +value. The string is enclosed in double quotes if the string contains +a single quote and no double quotes, else it's enclosed in single +quotes. (The \keyword{print} statement, described later, can be used +to write strings without quotes or escapes.) + +Strings can be concatenated (glued together) with the +\code{+} operator, and repeated with \code{*}: + +\begin{verbatim} +>>> word = 'Help' + 'A' +>>> word +'HelpA' +>>> '<' + word*5 + '>' +'<HelpAHelpAHelpAHelpAHelpA>' +\end{verbatim} + +Two string literals next to each other are automatically concatenated; +the first line above could also have been written \samp{word = 'Help' +'A'}; this only works with two literals, not with arbitrary string +expressions: + +\begin{verbatim} +>>> 'str' 'ing' # <- This is ok +'string' +>>> 'str'.strip() + 'ing' # <- This is ok +'string' +>>> 'str'.strip() 'ing' # <- This is invalid + File "<stdin>", line 1, in ? + 'str'.strip() 'ing' + ^ +SyntaxError: invalid syntax +\end{verbatim} + +Strings can be subscripted (indexed); like in C, the first character +of a string has subscript (index) 0. There is no separate character +type; a character is simply a string of size one. Like in Icon, +substrings can be specified with the \emph{slice notation}: two indices +separated by a colon. + +\begin{verbatim} +>>> word[4] +'A' +>>> word[0:2] +'He' +>>> word[2:4] +'lp' +\end{verbatim} + +Slice indices have useful defaults; an omitted first index defaults to +zero, an omitted second index defaults to the size of the string being +sliced. + +\begin{verbatim} +>>> word[:2] # The first two characters +'He' +>>> word[2:] # Everything except the first two characters +'lpA' +\end{verbatim} + +Unlike a C string, Python strings cannot be changed. Assigning to an +indexed position in the string results in an error: + +\begin{verbatim} +>>> word[0] = 'x' +Traceback (most recent call last): + File "<stdin>", line 1, in ? +TypeError: object doesn't support item assignment +>>> word[:1] = 'Splat' +Traceback (most recent call last): + File "<stdin>", line 1, in ? +TypeError: object doesn't support slice assignment +\end{verbatim} + +However, creating a new string with the combined content is easy and +efficient: + +\begin{verbatim} +>>> 'x' + word[1:] +'xelpA' +>>> 'Splat' + word[4] +'SplatA' +\end{verbatim} + +Here's a useful invariant of slice operations: +\code{s[:i] + s[i:]} equals \code{s}. + +\begin{verbatim} +>>> word[:2] + word[2:] +'HelpA' +>>> word[:3] + word[3:] +'HelpA' +\end{verbatim} + +Degenerate slice indices are handled gracefully: an index that is too +large is replaced by the string size, an upper bound smaller than the +lower bound returns an empty string. + +\begin{verbatim} +>>> word[1:100] +'elpA' +>>> word[10:] +'' +>>> word[2:1] +'' +\end{verbatim} + +Indices may be negative numbers, to start counting from the right. +For example: + +\begin{verbatim} +>>> word[-1] # The last character +'A' +>>> word[-2] # The last-but-one character +'p' +>>> word[-2:] # The last two characters +'pA' +>>> word[:-2] # Everything except the last two characters +'Hel' +\end{verbatim} + +But note that -0 is really the same as 0, so it does not count from +the right! + +\begin{verbatim} +>>> word[-0] # (since -0 equals 0) +'H' +\end{verbatim} + +Out-of-range negative slice indices are truncated, but don't try this +for single-element (non-slice) indices: + +\begin{verbatim} +>>> word[-100:] +'HelpA' +>>> word[-10] # error +Traceback (most recent call last): + File "<stdin>", line 1, in ? +IndexError: string index out of range +\end{verbatim} + +The best way to remember how slices work is to think of the indices as +pointing \emph{between} characters, with the left edge of the first +character numbered 0. Then the right edge of the last character of a +string of \var{n} characters has index \var{n}, for example: + +\begin{verbatim} + +---+---+---+---+---+ + | H | e | l | p | A | + +---+---+---+---+---+ + 0 1 2 3 4 5 +-5 -4 -3 -2 -1 +\end{verbatim} + +The first row of numbers gives the position of the indices 0...5 in +the string; the second row gives the corresponding negative indices. +The slice from \var{i} to \var{j} consists of all characters between +the edges labeled \var{i} and \var{j}, respectively. + +For non-negative indices, the length of a slice is the difference of +the indices, if both are within bounds. For example, the length of +\code{word[1:3]} is 2. + +The built-in function \function{len()} returns the length of a string: + +\begin{verbatim} +>>> s = 'supercalifragilisticexpialidocious' +>>> len(s) +34 +\end{verbatim} + + +\begin{seealso} + \seetitle[../lib/typesseq.html]{Sequence Types}% + {Strings, and the Unicode strings described in the next + section, are examples of \emph{sequence types}, and + support the common operations supported by such types.} + \seetitle[../lib/string-methods.html]{String Methods}% + {Both strings and Unicode strings support a large number of + methods for basic transformations and searching.} + \seetitle[../lib/typesseq-strings.html]{String Formatting Operations}% + {The formatting operations invoked when strings and Unicode + strings are the left operand of the \code{\%} operator are + described in more detail here.} +\end{seealso} + + +\subsection{Unicode Strings \label{unicodeStrings}} +\sectionauthor{Marc-Andre Lemburg}{mal@lemburg.com} + +Starting with Python 2.0 a new data type for storing text data is +available to the programmer: the Unicode object. It can be used to +store and manipulate Unicode data (see \url{http://www.unicode.org/}) +and integrates well with the existing string objects, providing +auto-conversions where necessary. + +Unicode has the advantage of providing one ordinal for every character +in every script used in modern and ancient texts. Previously, there +were only 256 possible ordinals for script characters. Texts were +typically bound to a code page which mapped the ordinals to script +characters. This lead to very much confusion especially with respect +to internationalization (usually written as \samp{i18n} --- +\character{i} + 18 characters + \character{n}) of software. Unicode +solves these problems by defining one code page for all scripts. + +Creating Unicode strings in Python is just as simple as creating +normal strings: + +\begin{verbatim} +>>> u'Hello World !' +u'Hello World !' +\end{verbatim} + +The small \character{u} in front of the quote indicates that a +Unicode string is supposed to be created. If you want to include +special characters in the string, you can do so by using the Python +\emph{Unicode-Escape} encoding. The following example shows how: + +\begin{verbatim} +>>> u'Hello\u0020World !' +u'Hello World !' +\end{verbatim} + +The escape sequence \code{\e u0020} indicates to insert the Unicode +character with the ordinal value 0x0020 (the space character) at the +given position. + +Other characters are interpreted by using their respective ordinal +values directly as Unicode ordinals. If you have literal strings +in the standard Latin-1 encoding that is used in many Western countries, +you will find it convenient that the lower 256 characters +of Unicode are the same as the 256 characters of Latin-1. + +For experts, there is also a raw mode just like the one for normal +strings. You have to prefix the opening quote with 'ur' to have +Python use the \emph{Raw-Unicode-Escape} encoding. It will only apply +the above \code{\e uXXXX} conversion if there is an uneven number of +backslashes in front of the small 'u'. + +\begin{verbatim} +>>> ur'Hello\u0020World !' +u'Hello World !' +>>> ur'Hello\\u0020World !' +u'Hello\\\\u0020World !' +\end{verbatim} + +The raw mode is most useful when you have to enter lots of +backslashes, as can be necessary in regular expressions. + +Apart from these standard encodings, Python provides a whole set of +other ways of creating Unicode strings on the basis of a known +encoding. + +The built-in function \function{unicode()}\bifuncindex{unicode} provides +access to all registered Unicode codecs (COders and DECoders). Some of +the more well known encodings which these codecs can convert are +\emph{Latin-1}, \emph{ASCII}, \emph{UTF-8}, and \emph{UTF-16}. +The latter two are variable-length encodings that store each Unicode +character in one or more bytes. The default encoding is +normally set to \ASCII, which passes through characters in the range +0 to 127 and rejects any other characters with an error. +When a Unicode string is printed, written to a file, or converted +with \function{str()}, conversion takes place using this default encoding. + +\begin{verbatim} +>>> u"abc" +u'abc' +>>> str(u"abc") +'abc' +>>> u"äöü" +u'\xe4\xf6\xfc' +>>> str(u"äöü") +Traceback (most recent call last): + File "<stdin>", line 1, in ? +UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-2: ordinal not in range(128) +\end{verbatim} + +To convert a Unicode string into an 8-bit string using a specific +encoding, Unicode objects provide an \function{encode()} method +that takes one argument, the name of the encoding. Lowercase names +for encodings are preferred. + +\begin{verbatim} +>>> u"äöü".encode('utf-8') +'\xc3\xa4\xc3\xb6\xc3\xbc' +\end{verbatim} + +If you have data in a specific encoding and want to produce a +corresponding Unicode string from it, you can use the +\function{unicode()} function with the encoding name as the second +argument. + +\begin{verbatim} +>>> unicode('\xc3\xa4\xc3\xb6\xc3\xbc', 'utf-8') +u'\xe4\xf6\xfc' +\end{verbatim} + +\subsection{Lists \label{lists}} + +Python knows a number of \emph{compound} data types, used to group +together other values. The most versatile is the \emph{list}, which +can be written as a list of comma-separated values (items) between +square brackets. List items need not all have the same type. + +\begin{verbatim} +>>> a = ['spam', 'eggs', 100, 1234] +>>> a +['spam', 'eggs', 100, 1234] +\end{verbatim} + +Like string indices, list indices start at 0, and lists can be sliced, +concatenated and so on: + +\begin{verbatim} +>>> a[0] +'spam' +>>> a[3] +1234 +>>> a[-2] +100 +>>> a[1:-1] +['eggs', 100] +>>> a[:2] + ['bacon', 2*2] +['spam', 'eggs', 'bacon', 4] +>>> 3*a[:3] + ['Boo!'] +['spam', 'eggs', 100, 'spam', 'eggs', 100, 'spam', 'eggs', 100, 'Boo!'] +\end{verbatim} + +Unlike strings, which are \emph{immutable}, it is possible to change +individual elements of a list: + +\begin{verbatim} +>>> a +['spam', 'eggs', 100, 1234] +>>> a[2] = a[2] + 23 +>>> a +['spam', 'eggs', 123, 1234] +\end{verbatim} + +Assignment to slices is also possible, and this can even change the size +of the list or clear it entirely: + +\begin{verbatim} +>>> # Replace some items: +... a[0:2] = [1, 12] +>>> a +[1, 12, 123, 1234] +>>> # Remove some: +... a[0:2] = [] +>>> a +[123, 1234] +>>> # Insert some: +... a[1:1] = ['bletch', 'xyzzy'] +>>> a +[123, 'bletch', 'xyzzy', 1234] +>>> # Insert (a copy of) itself at the beginning +>>> a[:0] = a +>>> a +[123, 'bletch', 'xyzzy', 1234, 123, 'bletch', 'xyzzy', 1234] +>>> # Clear the list: replace all items with an empty list +>>> a[:] = [] +>>> a +[] +\end{verbatim} + +The built-in function \function{len()} also applies to lists: + +\begin{verbatim} +>>> len(a) +8 +\end{verbatim} + +It is possible to nest lists (create lists containing other lists), +for example: + +\begin{verbatim} +>>> q = [2, 3] +>>> p = [1, q, 4] +>>> len(p) +3 +>>> p[1] +[2, 3] +>>> p[1][0] +2 +>>> p[1].append('xtra') # See section 5.1 +>>> p +[1, [2, 3, 'xtra'], 4] +>>> q +[2, 3, 'xtra'] +\end{verbatim} + +Note that in the last example, \code{p[1]} and \code{q} really refer to +the same object! We'll come back to \emph{object semantics} later. + +\section{First Steps Towards Programming \label{firstSteps}} + +Of course, we can use Python for more complicated tasks than adding +two and two together. For instance, we can write an initial +sub-sequence of the \emph{Fibonacci} series as follows: + +\begin{verbatim} +>>> # Fibonacci series: +... # the sum of two elements defines the next +... a, b = 0, 1 +>>> while b < 10: +... print b +... a, b = b, a+b +... +1 +1 +2 +3 +5 +8 +\end{verbatim} + +This example introduces several new features. + +\begin{itemize} + +\item +The first line contains a \emph{multiple assignment}: the variables +\code{a} and \code{b} simultaneously get the new values 0 and 1. On the +last line this is used again, demonstrating that the expressions on +the right-hand side are all evaluated first before any of the +assignments take place. The right-hand side expressions are evaluated +from the left to the right. + +\item +The \keyword{while} loop executes as long as the condition (here: +\code{b < 10}) remains true. In Python, like in C, any non-zero +integer value is true; zero is false. The condition may also be a +string or list value, in fact any sequence; anything with a non-zero +length is true, empty sequences are false. The test used in the +example is a simple comparison. The standard comparison operators are +written the same as in C: \code{<} (less than), \code{>} (greater than), +\code{==} (equal to), \code{<=} (less than or equal to), +\code{>=} (greater than or equal to) and \code{!=} (not equal to). + +\item +The \emph{body} of the loop is \emph{indented}: indentation is Python's +way of grouping statements. Python does not (yet!) provide an +intelligent input line editing facility, so you have to type a tab or +space(s) for each indented line. In practice you will prepare more +complicated input for Python with a text editor; most text editors have +an auto-indent facility. When a compound statement is entered +interactively, it must be followed by a blank line to indicate +completion (since the parser cannot guess when you have typed the last +line). Note that each line within a basic block must be indented by +the same amount. + +\item +The \keyword{print} statement writes the value of the expression(s) it is +given. It differs from just writing the expression you want to write +(as we did earlier in the calculator examples) in the way it handles +multiple expressions and strings. Strings are printed without quotes, +and a space is inserted between items, so you can format things nicely, +like this: + +\begin{verbatim} +>>> i = 256*256 +>>> print 'The value of i is', i +The value of i is 65536 +\end{verbatim} + +A trailing comma avoids the newline after the output: + +\begin{verbatim} +>>> a, b = 0, 1 +>>> while b < 1000: +... print b, +... a, b = b, a+b +... +1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987 +\end{verbatim} + +Note that the interpreter inserts a newline before it prints the next +prompt if the last line was not completed. + +\end{itemize} + + +\chapter{More Control Flow Tools \label{moreControl}} + +Besides the \keyword{while} statement just introduced, Python knows +the usual control flow statements known from other languages, with +some twists. + +\section{\keyword{if} Statements \label{if}} + +Perhaps the most well-known statement type is the +\keyword{if} statement. For example: + +\begin{verbatim} +>>> x = int(raw_input("Please enter an integer: ")) +>>> if x < 0: +... x = 0 +... print 'Negative changed to zero' +... elif x == 0: +... print 'Zero' +... elif x == 1: +... print 'Single' +... else: +... print 'More' +... +\end{verbatim} + +There can be zero or more \keyword{elif} parts, and the +\keyword{else} part is optional. The keyword `\keyword{elif}' is +short for `else if', and is useful to avoid excessive indentation. An +\keyword{if} \ldots\ \keyword{elif} \ldots\ \keyword{elif} \ldots\ sequence +% Weird spacings happen here if the wrapping of the source text +% gets changed in the wrong way. +is a substitute for the \keyword{switch} or +\keyword{case} statements found in other languages. + + +\section{\keyword{for} Statements \label{for}} + +The \keyword{for}\stindex{for} statement in Python differs a bit from +what you may be used to in C or Pascal. Rather than always +iterating over an arithmetic progression of numbers (like in Pascal), +or giving the user the ability to define both the iteration step and +halting condition (as C), Python's +\keyword{for}\stindex{for} statement iterates over the items of any +sequence (a list or a string), in the order that they appear in +the sequence. For example (no pun intended): +% One suggestion was to give a real C example here, but that may only +% serve to confuse non-C programmers. + +\begin{verbatim} +>>> # Measure some strings: +... a = ['cat', 'window', 'defenestrate'] +>>> for x in a: +... print x, len(x) +... +cat 3 +window 6 +defenestrate 12 +\end{verbatim} + +It is not safe to modify the sequence being iterated over in the loop +(this can only happen for mutable sequence types, such as lists). If +you need to modify the list you are iterating over (for example, to +duplicate selected items) you must iterate over a copy. The slice +notation makes this particularly convenient: + +\begin{verbatim} +>>> for x in a[:]: # make a slice copy of the entire list +... if len(x) > 6: a.insert(0, x) +... +>>> a +['defenestrate', 'cat', 'window', 'defenestrate'] +\end{verbatim} + + +\section{The \function{range()} Function \label{range}} + +If you do need to iterate over a sequence of numbers, the built-in +function \function{range()} comes in handy. It generates lists +containing arithmetic progressions: + +\begin{verbatim} +>>> range(10) +[0, 1, 2, 3, 4, 5, 6, 7, 8, 9] +\end{verbatim} + +The given end point is never part of the generated list; +\code{range(10)} generates a list of 10 values, the legal +indices for items of a sequence of length 10. It is possible to let +the range start at another number, or to specify a different increment +(even negative; sometimes this is called the `step'): + +\begin{verbatim} +>>> range(5, 10) +[5, 6, 7, 8, 9] +>>> range(0, 10, 3) +[0, 3, 6, 9] +>>> range(-10, -100, -30) +[-10, -40, -70] +\end{verbatim} + +To iterate over the indices of a sequence, combine +\function{range()} and \function{len()} as follows: + +\begin{verbatim} +>>> a = ['Mary', 'had', 'a', 'little', 'lamb'] +>>> for i in range(len(a)): +... print i, a[i] +... +0 Mary +1 had +2 a +3 little +4 lamb +\end{verbatim} + + +\section{\keyword{break} and \keyword{continue} Statements, and + \keyword{else} Clauses on Loops + \label{break}} + +The \keyword{break} statement, like in C, breaks out of the smallest +enclosing \keyword{for} or \keyword{while} loop. + +The \keyword{continue} statement, also borrowed from C, continues +with the next iteration of the loop. + +Loop statements may have an \code{else} clause; it is executed when +the loop terminates through exhaustion of the list (with +\keyword{for}) or when the condition becomes false (with +\keyword{while}), but not when the loop is terminated by a +\keyword{break} statement. This is exemplified by the following loop, +which searches for prime numbers: + +\begin{verbatim} +>>> for n in range(2, 10): +... for x in range(2, n): +... if n % x == 0: +... print n, 'equals', x, '*', n/x +... break +... else: +... # loop fell through without finding a factor +... print n, 'is a prime number' +... +2 is a prime number +3 is a prime number +4 equals 2 * 2 +5 is a prime number +6 equals 2 * 3 +7 is a prime number +8 equals 2 * 4 +9 equals 3 * 3 +\end{verbatim} + + +\section{\keyword{pass} Statements \label{pass}} + +The \keyword{pass} statement does nothing. +It can be used when a statement is required syntactically but the +program requires no action. +For example: + +\begin{verbatim} +>>> while True: +... pass # Busy-wait for keyboard interrupt +... +\end{verbatim} + + +\section{Defining Functions \label{functions}} + +We can create a function that writes the Fibonacci series to an +arbitrary boundary: + +\begin{verbatim} +>>> def fib(n): # write Fibonacci series up to n +... """Print a Fibonacci series up to n.""" +... a, b = 0, 1 +... while b < n: +... print b, +... a, b = b, a+b +... +>>> # Now call the function we just defined: +... fib(2000) +1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987 1597 +\end{verbatim} + +The keyword \keyword{def} introduces a function \emph{definition}. It +must be followed by the function name and the parenthesized list of +formal parameters. The statements that form the body of the function +start at the next line, and must be indented. The first statement of +the function body can optionally be a string literal; this string +literal is the function's \index{documentation strings}documentation +string, or \dfn{docstring}.\index{docstrings}\index{strings, documentation} + +There are tools which use docstrings to automatically produce online +or printed documentation, or to let the user interactively browse +through code; it's good practice to include docstrings in code that +you write, so try to make a habit of it. + +The \emph{execution} of a function introduces a new symbol table used +for the local variables of the function. More precisely, all variable +assignments in a function store the value in the local symbol table; +whereas variable references first look in the local symbol table, then +in the global symbol table, and then in the table of built-in names. +Thus, global variables cannot be directly assigned a value within a +function (unless named in a \keyword{global} statement), although +they may be referenced. + +The actual parameters (arguments) to a function call are introduced in +the local symbol table of the called function when it is called; thus, +arguments are passed using \emph{call by value} (where the +\emph{value} is always an object \emph{reference}, not the value of +the object).\footnote{ + Actually, \emph{call by object reference} would be a better + description, since if a mutable object is passed, the caller + will see any changes the callee makes to it (items + inserted into a list). +} When a function calls another function, a new local symbol table is +created for that call. + +A function definition introduces the function name in the current +symbol table. The value of the function name +has a type that is recognized by the interpreter as a user-defined +function. This value can be assigned to another name which can then +also be used as a function. This serves as a general renaming +mechanism: + +\begin{verbatim} +>>> fib +<function fib at 10042ed0> +>>> f = fib +>>> f(100) +1 1 2 3 5 8 13 21 34 55 89 +\end{verbatim} + +You might object that \code{fib} is not a function but a procedure. In +Python, like in C, procedures are just functions that don't return a +value. In fact, technically speaking, procedures do return a value, +albeit a rather boring one. This value is called \code{None} (it's a +built-in name). Writing the value \code{None} is normally suppressed by +the interpreter if it would be the only value written. You can see it +if you really want to: + +\begin{verbatim} +>>> print fib(0) +None +\end{verbatim} + +It is simple to write a function that returns a list of the numbers of +the Fibonacci series, instead of printing it: + +\begin{verbatim} +>>> def fib2(n): # return Fibonacci series up to n +... """Return a list containing the Fibonacci series up to n.""" +... result = [] +... a, b = 0, 1 +... while b < n: +... result.append(b) # see below +... a, b = b, a+b +... return result +... +>>> f100 = fib2(100) # call it +>>> f100 # write the result +[1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89] +\end{verbatim} + +This example, as usual, demonstrates some new Python features: + +\begin{itemize} + +\item +The \keyword{return} statement returns with a value from a function. +\keyword{return} without an expression argument returns \code{None}. +Falling off the end of a procedure also returns \code{None}. + +\item +The statement \code{result.append(b)} calls a \emph{method} of the list +object \code{result}. A method is a function that `belongs' to an +object and is named \code{obj.methodname}, where \code{obj} is some +object (this may be an expression), and \code{methodname} is the name +of a method that is defined by the object's type. Different types +define different methods. Methods of different types may have the +same name without causing ambiguity. (It is possible to define your +own object types and methods, using \emph{classes}, as discussed later +in this tutorial.) +The method \method{append()} shown in the example is defined for +list objects; it adds a new element at the end of the list. In this +example it is equivalent to \samp{result = result + [b]}, but more +efficient. + +\end{itemize} + +\section{More on Defining Functions \label{defining}} + +It is also possible to define functions with a variable number of +arguments. There are three forms, which can be combined. + +\subsection{Default Argument Values \label{defaultArgs}} + +The most useful form is to specify a default value for one or more +arguments. This creates a function that can be called with fewer +arguments than it is defined to allow. For example: + +\begin{verbatim} +def ask_ok(prompt, retries=4, complaint='Yes or no, please!'): + while True: + ok = raw_input(prompt) + if ok in ('y', 'ye', 'yes'): return True + if ok in ('n', 'no', 'nop', 'nope'): return False + retries = retries - 1 + if retries < 0: raise IOError, 'refusenik user' + print complaint +\end{verbatim} + +This function can be called either like this: +\code{ask_ok('Do you really want to quit?')} or like this: +\code{ask_ok('OK to overwrite the file?', 2)}. + +This example also introduces the \keyword{in} keyword. This tests +whether or not a sequence contains a certain value. + +The default values are evaluated at the point of function definition +in the \emph{defining} scope, so that + +\begin{verbatim} +i = 5 + +def f(arg=i): + print arg + +i = 6 +f() +\end{verbatim} + +will print \code{5}. + +\strong{Important warning:} The default value is evaluated only once. +This makes a difference when the default is a mutable object such as a +list, dictionary, or instances of most classes. For example, the +following function accumulates the arguments passed to it on +subsequent calls: + +\begin{verbatim} +def f(a, L=[]): + L.append(a) + return L + +print f(1) +print f(2) +print f(3) +\end{verbatim} + +This will print + +\begin{verbatim} +[1] +[1, 2] +[1, 2, 3] +\end{verbatim} + +If you don't want the default to be shared between subsequent calls, +you can write the function like this instead: + +\begin{verbatim} +def f(a, L=None): + if L is None: + L = [] + L.append(a) + return L +\end{verbatim} + +\subsection{Keyword Arguments \label{keywordArgs}} + +Functions can also be called using +keyword arguments of the form \samp{\var{keyword} = \var{value}}. For +instance, the following function: + +\begin{verbatim} +def parrot(voltage, state='a stiff', action='voom', type='Norwegian Blue'): + print "-- This parrot wouldn't", action, + print "if you put", voltage, "volts through it." + print "-- Lovely plumage, the", type + print "-- It's", state, "!" +\end{verbatim} + +could be called in any of the following ways: + +\begin{verbatim} +parrot(1000) +parrot(action = 'VOOOOOM', voltage = 1000000) +parrot('a thousand', state = 'pushing up the daisies') +parrot('a million', 'bereft of life', 'jump') +\end{verbatim} + +but the following calls would all be invalid: + +\begin{verbatim} +parrot() # required argument missing +parrot(voltage=5.0, 'dead') # non-keyword argument following keyword +parrot(110, voltage=220) # duplicate value for argument +parrot(actor='John Cleese') # unknown keyword +\end{verbatim} + +In general, an argument list must have any positional arguments +followed by any keyword arguments, where the keywords must be chosen +from the formal parameter names. It's not important whether a formal +parameter has a default value or not. No argument may receive a +value more than once --- formal parameter names corresponding to +positional arguments cannot be used as keywords in the same calls. +Here's an example that fails due to this restriction: + +\begin{verbatim} +>>> def function(a): +... pass +... +>>> function(0, a=0) +Traceback (most recent call last): + File "<stdin>", line 1, in ? +TypeError: function() got multiple values for keyword argument 'a' +\end{verbatim} + +When a final formal parameter of the form \code{**\var{name}} is +present, it receives a \ulink{dictionary}{../lib/typesmapping.html} +containing all keyword arguments except for those corresponding to +a formal parameter. This may be +combined with a formal parameter of the form +\code{*\var{name}} (described in the next subsection) which receives a +tuple containing the positional arguments beyond the formal parameter +list. (\code{*\var{name}} must occur before \code{**\var{name}}.) +For example, if we define a function like this: + +\begin{verbatim} +def cheeseshop(kind, *arguments, **keywords): + print "-- Do you have any", kind, '?' + print "-- I'm sorry, we're all out of", kind + for arg in arguments: print arg + print '-'*40 + keys = keywords.keys() + keys.sort() + for kw in keys: print kw, ':', keywords[kw] +\end{verbatim} + +It could be called like this: + +\begin{verbatim} +cheeseshop('Limburger', "It's very runny, sir.", + "It's really very, VERY runny, sir.", + client='John Cleese', + shopkeeper='Michael Palin', + sketch='Cheese Shop Sketch') +\end{verbatim} + +and of course it would print: + +\begin{verbatim} +-- Do you have any Limburger ? +-- I'm sorry, we're all out of Limburger +It's very runny, sir. +It's really very, VERY runny, sir. +---------------------------------------- +client : John Cleese +shopkeeper : Michael Palin +sketch : Cheese Shop Sketch +\end{verbatim} + +Note that the \method{sort()} method of the list of keyword argument +names is called before printing the contents of the \code{keywords} +dictionary; if this is not done, the order in which the arguments are +printed is undefined. + + +\subsection{Arbitrary Argument Lists \label{arbitraryArgs}} + +Finally, the least frequently used option is to specify that a +function can be called with an arbitrary number of arguments. These +arguments will be wrapped up in a tuple. Before the variable number +of arguments, zero or more normal arguments may occur. + +\begin{verbatim} +def fprintf(file, format, *args): + file.write(format % args) +\end{verbatim} + + +\subsection{Unpacking Argument Lists \label{unpacking-arguments}} + +The reverse situation occurs when the arguments are already in a list +or tuple but need to be unpacked for a function call requiring separate +positional arguments. For instance, the built-in \function{range()} +function expects separate \var{start} and \var{stop} arguments. If they +are not available separately, write the function call with the +\code{*}-operator to unpack the arguments out of a list or tuple: + +\begin{verbatim} +>>> range(3, 6) # normal call with separate arguments +[3, 4, 5] +>>> args = [3, 6] +>>> range(*args) # call with arguments unpacked from a list +[3, 4, 5] +\end{verbatim} + +In the same fashion, dictionaries can deliver keyword arguments with the +\code{**}-operator: + +\begin{verbatim} +>>> def parrot(voltage, state='a stiff', action='voom'): +... print "-- This parrot wouldn't", action, +... print "if you put", voltage, "volts through it.", +... print "E's", state, "!" +... +>>> d = {"voltage": "four million", "state": "bleedin' demised", "action": "VOOM"} +>>> parrot(**d) +-- This parrot wouldn't VOOM if you put four million volts through it. E's bleedin' demised ! +\end{verbatim} + + +\subsection{Lambda Forms \label{lambda}} + +By popular demand, a few features commonly found in functional +programming languages like Lisp have been added to Python. With the +\keyword{lambda} keyword, small anonymous functions can be created. +Here's a function that returns the sum of its two arguments: +\samp{lambda a, b: a+b}. Lambda forms can be used wherever function +objects are required. They are syntactically restricted to a single +expression. Semantically, they are just syntactic sugar for a normal +function definition. Like nested function definitions, lambda forms +can reference variables from the containing scope: + +\begin{verbatim} +>>> def make_incrementor(n): +... return lambda x: x + n +... +>>> f = make_incrementor(42) +>>> f(0) +42 +>>> f(1) +43 +\end{verbatim} + + +\subsection{Documentation Strings \label{docstrings}} + +There are emerging conventions about the content and formatting of +documentation strings. +\index{docstrings}\index{documentation strings} +\index{strings, documentation} + +The first line should always be a short, concise summary of the +object's purpose. For brevity, it should not explicitly state the +object's name or type, since these are available by other means +(except if the name happens to be a verb describing a function's +operation). This line should begin with a capital letter and end with +a period. + +If there are more lines in the documentation string, the second line +should be blank, visually separating the summary from the rest of the +description. The following lines should be one or more paragraphs +describing the object's calling conventions, its side effects, etc. + +The Python parser does not strip indentation from multi-line string +literals in Python, so tools that process documentation have to strip +indentation if desired. This is done using the following convention. +The first non-blank line \emph{after} the first line of the string +determines the amount of indentation for the entire documentation +string. (We can't use the first line since it is generally adjacent +to the string's opening quotes so its indentation is not apparent in +the string literal.) Whitespace ``equivalent'' to this indentation is +then stripped from the start of all lines of the string. Lines that +are indented less should not occur, but if they occur all their +leading whitespace should be stripped. Equivalence of whitespace +should be tested after expansion of tabs (to 8 spaces, normally). + +Here is an example of a multi-line docstring: + +\begin{verbatim} +>>> def my_function(): +... """Do nothing, but document it. +... +... No, really, it doesn't do anything. +... """ +... pass +... +>>> print my_function.__doc__ +Do nothing, but document it. + + No, really, it doesn't do anything. + +\end{verbatim} + + + +\chapter{Data Structures \label{structures}} + +This chapter describes some things you've learned about already in +more detail, and adds some new things as well. + + +\section{More on Lists \label{moreLists}} + +The list data type has some more methods. Here are all of the methods +of list objects: + +\begin{methoddesc}[list]{append}{x} +Add an item to the end of the list; +equivalent to \code{a[len(a):] = [\var{x}]}. +\end{methoddesc} + +\begin{methoddesc}[list]{extend}{L} +Extend the list by appending all the items in the given list; +equivalent to \code{a[len(a):] = \var{L}}. +\end{methoddesc} + +\begin{methoddesc}[list]{insert}{i, x} +Insert an item at a given position. The first argument is the index +of the element before which to insert, so \code{a.insert(0, \var{x})} +inserts at the front of the list, and \code{a.insert(len(a), \var{x})} +is equivalent to \code{a.append(\var{x})}. +\end{methoddesc} + +\begin{methoddesc}[list]{remove}{x} +Remove the first item from the list whose value is \var{x}. +It is an error if there is no such item. +\end{methoddesc} + +\begin{methoddesc}[list]{pop}{\optional{i}} +Remove the item at the given position in the list, and return it. If +no index is specified, \code{a.pop()} removes and returns the last item +in the list. (The square brackets +around the \var{i} in the method signature denote that the parameter +is optional, not that you should type square brackets at that +position. You will see this notation frequently in the +\citetitle[../lib/lib.html]{Python Library Reference}.) +\end{methoddesc} + +\begin{methoddesc}[list]{index}{x} +Return the index in the list of the first item whose value is \var{x}. +It is an error if there is no such item. +\end{methoddesc} + +\begin{methoddesc}[list]{count}{x} +Return the number of times \var{x} appears in the list. +\end{methoddesc} + +\begin{methoddesc}[list]{sort}{} +Sort the items of the list, in place. +\end{methoddesc} + +\begin{methoddesc}[list]{reverse}{} +Reverse the elements of the list, in place. +\end{methoddesc} + +An example that uses most of the list methods: + +\begin{verbatim} +>>> a = [66.25, 333, 333, 1, 1234.5] +>>> print a.count(333), a.count(66.25), a.count('x') +2 1 0 +>>> a.insert(2, -1) +>>> a.append(333) +>>> a +[66.25, 333, -1, 333, 1, 1234.5, 333] +>>> a.index(333) +1 +>>> a.remove(333) +>>> a +[66.25, -1, 333, 1, 1234.5, 333] +>>> a.reverse() +>>> a +[333, 1234.5, 1, 333, -1, 66.25] +>>> a.sort() +>>> a +[-1, 1, 66.25, 333, 333, 1234.5] +\end{verbatim} + + +\subsection{Using Lists as Stacks \label{lists-as-stacks}} +\sectionauthor{Ka-Ping Yee}{ping@lfw.org} + +The list methods make it very easy to use a list as a stack, where the +last element added is the first element retrieved (``last-in, +first-out''). To add an item to the top of the stack, use +\method{append()}. To retrieve an item from the top of the stack, use +\method{pop()} without an explicit index. For example: + +\begin{verbatim} +>>> stack = [3, 4, 5] +>>> stack.append(6) +>>> stack.append(7) +>>> stack +[3, 4, 5, 6, 7] +>>> stack.pop() +7 +>>> stack +[3, 4, 5, 6] +>>> stack.pop() +6 +>>> stack.pop() +5 +>>> stack +[3, 4] +\end{verbatim} + + +\subsection{Using Lists as Queues \label{lists-as-queues}} +\sectionauthor{Ka-Ping Yee}{ping@lfw.org} + +You can also use a list conveniently as a queue, where the first +element added is the first element retrieved (``first-in, +first-out''). To add an item to the back of the queue, use +\method{append()}. To retrieve an item from the front of the queue, +use \method{pop()} with \code{0} as the index. For example: + +\begin{verbatim} +>>> queue = ["Eric", "John", "Michael"] +>>> queue.append("Terry") # Terry arrives +>>> queue.append("Graham") # Graham arrives +>>> queue.pop(0) +'Eric' +>>> queue.pop(0) +'John' +>>> queue +['Michael', 'Terry', 'Graham'] +\end{verbatim} + + +\subsection{Functional Programming Tools \label{functional}} + +There are three built-in functions that are very useful when used with +lists: \function{filter()}, \function{map()}, and \function{reduce()}. + +\samp{filter(\var{function}, \var{sequence})} returns a sequence +consisting of those items from the +sequence for which \code{\var{function}(\var{item})} is true. +If \var{sequence} is a \class{string} or \class{tuple}, the result will +be of the same type; otherwise, it is always a \class{list}. +For example, to compute some primes: + +\begin{verbatim} +>>> def f(x): return x % 2 != 0 and x % 3 != 0 +... +>>> filter(f, range(2, 25)) +[5, 7, 11, 13, 17, 19, 23] +\end{verbatim} + +\samp{map(\var{function}, \var{sequence})} calls +\code{\var{function}(\var{item})} for each of the sequence's items and +returns a list of the return values. For example, to compute some +cubes: + +\begin{verbatim} +>>> def cube(x): return x*x*x +... +>>> map(cube, range(1, 11)) +[1, 8, 27, 64, 125, 216, 343, 512, 729, 1000] +\end{verbatim} + +More than one sequence may be passed; the function must then have as +many arguments as there are sequences and is called with the +corresponding item from each sequence (or \code{None} if some sequence +is shorter than another). For example: + +\begin{verbatim} +>>> seq = range(8) +>>> def add(x, y): return x+y +... +>>> map(add, seq, seq) +[0, 2, 4, 6, 8, 10, 12, 14] +\end{verbatim} + +\samp{reduce(\var{function}, \var{sequence})} returns a single value +constructed by calling the binary function \var{function} on the first two +items of the sequence, then on the result and the next item, and so +on. For example, to compute the sum of the numbers 1 through 10: + +\begin{verbatim} +>>> def add(x,y): return x+y +... +>>> reduce(add, range(1, 11)) +55 +\end{verbatim} + +If there's only one item in the sequence, its value is returned; if +the sequence is empty, an exception is raised. + +A third argument can be passed to indicate the starting value. In this +case the starting value is returned for an empty sequence, and the +function is first applied to the starting value and the first sequence +item, then to the result and the next item, and so on. For example, + +\begin{verbatim} +>>> def sum(seq): +... def add(x,y): return x+y +... return reduce(add, seq, 0) +... +>>> sum(range(1, 11)) +55 +>>> sum([]) +0 +\end{verbatim} + +Don't use this example's definition of \function{sum()}: since summing +numbers is such a common need, a built-in function +\code{sum(\var{sequence})} is already provided, and works exactly like +this. +\versionadded{2.3} + +\subsection{List Comprehensions} + +List comprehensions provide a concise way to create lists without resorting +to use of \function{map()}, \function{filter()} and/or \keyword{lambda}. +The resulting list definition tends often to be clearer than lists built +using those constructs. Each list comprehension consists of an expression +followed by a \keyword{for} clause, then zero or more \keyword{for} or +\keyword{if} clauses. The result will be a list resulting from evaluating +the expression in the context of the \keyword{for} and \keyword{if} clauses +which follow it. If the expression would evaluate to a tuple, it must be +parenthesized. + +\begin{verbatim} +>>> freshfruit = [' banana', ' loganberry ', 'passion fruit '] +>>> [weapon.strip() for weapon in freshfruit] +['banana', 'loganberry', 'passion fruit'] +>>> vec = [2, 4, 6] +>>> [3*x for x in vec] +[6, 12, 18] +>>> [3*x for x in vec if x > 3] +[12, 18] +>>> [3*x for x in vec if x < 2] +[] +>>> [[x,x**2] for x in vec] +[[2, 4], [4, 16], [6, 36]] +>>> [x, x**2 for x in vec] # error - parens required for tuples + File "<stdin>", line 1, in ? + [x, x**2 for x in vec] + ^ +SyntaxError: invalid syntax +>>> [(x, x**2) for x in vec] +[(2, 4), (4, 16), (6, 36)] +>>> vec1 = [2, 4, 6] +>>> vec2 = [4, 3, -9] +>>> [x*y for x in vec1 for y in vec2] +[8, 6, -18, 16, 12, -36, 24, 18, -54] +>>> [x+y for x in vec1 for y in vec2] +[6, 5, -7, 8, 7, -5, 10, 9, -3] +>>> [vec1[i]*vec2[i] for i in range(len(vec1))] +[8, 12, -54] +\end{verbatim} + +List comprehensions are much more flexible than \function{map()} and can be +applied to complex expressions and nested functions: + +\begin{verbatim} +>>> [str(round(355/113.0, i)) for i in range(1,6)] +['3.1', '3.14', '3.142', '3.1416', '3.14159'] +\end{verbatim} + + +\section{The \keyword{del} statement \label{del}} + +There is a way to remove an item from a list given its index instead +of its value: the \keyword{del} statement. This differs from the +\method{pop()} method which returns a value. The \keyword{del} +statement can also be used to remove slices from a list or clear the +entire list (which we did earlier by assignment of an empty list to +the slice). For example: + +\begin{verbatim} +>>> a = [-1, 1, 66.25, 333, 333, 1234.5] +>>> del a[0] +>>> a +[1, 66.25, 333, 333, 1234.5] +>>> del a[2:4] +>>> a +[1, 66.25, 1234.5] +>>> del a[:] +>>> a +[] +\end{verbatim} + +\keyword{del} can also be used to delete entire variables: + +\begin{verbatim} +>>> del a +\end{verbatim} + +Referencing the name \code{a} hereafter is an error (at least until +another value is assigned to it). We'll find other uses for +\keyword{del} later. + + +\section{Tuples and Sequences \label{tuples}} + +We saw that lists and strings have many common properties, such as +indexing and slicing operations. They are two examples of +\ulink{\emph{sequence} data types}{../lib/typesseq.html}. Since +Python is an evolving language, other sequence data types may be +added. There is also another standard sequence data type: the +\emph{tuple}. + +A tuple consists of a number of values separated by commas, for +instance: + +\begin{verbatim} +>>> t = 12345, 54321, 'hello!' +>>> t[0] +12345 +>>> t +(12345, 54321, 'hello!') +>>> # Tuples may be nested: +... u = t, (1, 2, 3, 4, 5) +>>> u +((12345, 54321, 'hello!'), (1, 2, 3, 4, 5)) +\end{verbatim} + +As you see, on output tuples are always enclosed in parentheses, so +that nested tuples are interpreted correctly; they may be input with +or without surrounding parentheses, although often parentheses are +necessary anyway (if the tuple is part of a larger expression). + +Tuples have many uses. For example: (x, y) coordinate pairs, employee +records from a database, etc. Tuples, like strings, are immutable: it +is not possible to assign to the individual items of a tuple (you can +simulate much of the same effect with slicing and concatenation, +though). It is also possible to create tuples which contain mutable +objects, such as lists. + +A special problem is the construction of tuples containing 0 or 1 +items: the syntax has some extra quirks to accommodate these. Empty +tuples are constructed by an empty pair of parentheses; a tuple with +one item is constructed by following a value with a comma +(it is not sufficient to enclose a single value in parentheses). +Ugly, but effective. For example: + +\begin{verbatim} +>>> empty = () +>>> singleton = 'hello', # <-- note trailing comma +>>> len(empty) +0 +>>> len(singleton) +1 +>>> singleton +('hello',) +\end{verbatim} + +The statement \code{t = 12345, 54321, 'hello!'} is an example of +\emph{tuple packing}: the values \code{12345}, \code{54321} and +\code{'hello!'} are packed together in a tuple. The reverse operation +is also possible: + +\begin{verbatim} +>>> x, y, z = t +\end{verbatim} + +This is called, appropriately enough, \emph{sequence unpacking}. +Sequence unpacking requires the list of variables on the left to +have the same number of elements as the length of the sequence. Note +that multiple assignment is really just a combination of tuple packing +and sequence unpacking! + +There is a small bit of asymmetry here: packing multiple values +always creates a tuple, and unpacking works for any sequence. + +% XXX Add a bit on the difference between tuples and lists. + + +\section{Sets \label{sets}} + +Python also includes a data type for \emph{sets}. A set is an unordered +collection with no duplicate elements. Basic uses include membership +testing and eliminating duplicate entries. Set objects also support +mathematical operations like union, intersection, difference, and +symmetric difference. + +Here is a brief demonstration: + +\begin{verbatim} +>>> basket = ['apple', 'orange', 'apple', 'pear', 'orange', 'banana'] +>>> fruit = set(basket) # create a set without duplicates +>>> fruit +set(['orange', 'pear', 'apple', 'banana']) +>>> 'orange' in fruit # fast membership testing +True +>>> 'crabgrass' in fruit +False + +>>> # Demonstrate set operations on unique letters from two words +... +>>> a = set('abracadabra') +>>> b = set('alacazam') +>>> a # unique letters in a +set(['a', 'r', 'b', 'c', 'd']) +>>> a - b # letters in a but not in b +set(['r', 'd', 'b']) +>>> a | b # letters in either a or b +set(['a', 'c', 'r', 'd', 'b', 'm', 'z', 'l']) +>>> a & b # letters in both a and b +set(['a', 'c']) +>>> a ^ b # letters in a or b but not both +set(['r', 'd', 'b', 'm', 'z', 'l']) +\end{verbatim} + + +\section{Dictionaries \label{dictionaries}} + +Another useful data type built into Python is the +\ulink{\emph{dictionary}}{../lib/typesmapping.html}. +Dictionaries are sometimes found in other languages as ``associative +memories'' or ``associative arrays''. Unlike sequences, which are +indexed by a range of numbers, dictionaries are indexed by \emph{keys}, +which can be any immutable type; strings and numbers can always be +keys. Tuples can be used as keys if they contain only strings, +numbers, or tuples; if a tuple contains any mutable object either +directly or indirectly, it cannot be used as a key. You can't use +lists as keys, since lists can be modified in place using +index assignments, slice assignments, or methods like +\method{append()} and \method{extend()}. + +It is best to think of a dictionary as an unordered set of +\emph{key: value} pairs, with the requirement that the keys are unique +(within one dictionary). +A pair of braces creates an empty dictionary: \code{\{\}}. +Placing a comma-separated list of key:value pairs within the +braces adds initial key:value pairs to the dictionary; this is also the +way dictionaries are written on output. + +The main operations on a dictionary are storing a value with some key +and extracting the value given the key. It is also possible to delete +a key:value pair +with \code{del}. +If you store using a key that is already in use, the old value +associated with that key is forgotten. It is an error to extract a +value using a non-existent key. + +The \method{keys()} method of a dictionary object returns a list of all +the keys used in the dictionary, in arbitrary order (if you want it +sorted, just apply the \method{sort()} method to the list of keys). To +check whether a single key is in the dictionary, either use the dictionary's +\method{has_key()} method or the \keyword{in} keyword. + +Here is a small example using a dictionary: + +\begin{verbatim} +>>> tel = {'jack': 4098, 'sape': 4139} +>>> tel['guido'] = 4127 +>>> tel +{'sape': 4139, 'guido': 4127, 'jack': 4098} +>>> tel['jack'] +4098 +>>> del tel['sape'] +>>> tel['irv'] = 4127 +>>> tel +{'guido': 4127, 'irv': 4127, 'jack': 4098} +>>> tel.keys() +['guido', 'irv', 'jack'] +>>> tel.has_key('guido') +True +>>> 'guido' in tel +True +\end{verbatim} + +The \function{dict()} constructor builds dictionaries directly from +lists of key-value pairs stored as tuples. When the pairs form a +pattern, list comprehensions can compactly specify the key-value list. + +\begin{verbatim} +>>> dict([('sape', 4139), ('guido', 4127), ('jack', 4098)]) +{'sape': 4139, 'jack': 4098, 'guido': 4127} +>>> dict([(x, x**2) for x in (2, 4, 6)]) # use a list comprehension +{2: 4, 4: 16, 6: 36} +\end{verbatim} + +Later in the tutorial, we will learn about Generator Expressions +which are even better suited for the task of supplying key-values pairs to +the \function{dict()} constructor. + +When the keys are simple strings, it is sometimes easier to specify +pairs using keyword arguments: + +\begin{verbatim} +>>> dict(sape=4139, guido=4127, jack=4098) +{'sape': 4139, 'jack': 4098, 'guido': 4127} +\end{verbatim} + + +\section{Looping Techniques \label{loopidioms}} + +When looping through dictionaries, the key and corresponding value can +be retrieved at the same time using the \method{iteritems()} method. + +\begin{verbatim} +>>> knights = {'gallahad': 'the pure', 'robin': 'the brave'} +>>> for k, v in knights.iteritems(): +... print k, v +... +gallahad the pure +robin the brave +\end{verbatim} + +When looping through a sequence, the position index and corresponding +value can be retrieved at the same time using the +\function{enumerate()} function. + +\begin{verbatim} +>>> for i, v in enumerate(['tic', 'tac', 'toe']): +... print i, v +... +0 tic +1 tac +2 toe +\end{verbatim} + +To loop over two or more sequences at the same time, the entries +can be paired with the \function{zip()} function. + +\begin{verbatim} +>>> questions = ['name', 'quest', 'favorite color'] +>>> answers = ['lancelot', 'the holy grail', 'blue'] +>>> for q, a in zip(questions, answers): +... print 'What is your %s? It is %s.' % (q, a) +... +What is your name? It is lancelot. +What is your quest? It is the holy grail. +What is your favorite color? It is blue. +\end{verbatim} + +To loop over a sequence in reverse, first specify the sequence +in a forward direction and then call the \function{reversed()} +function. + +\begin{verbatim} +>>> for i in reversed(xrange(1,10,2)): +... print i +... +9 +7 +5 +3 +1 +\end{verbatim} + +To loop over a sequence in sorted order, use the \function{sorted()} +function which returns a new sorted list while leaving the source +unaltered. + +\begin{verbatim} +>>> basket = ['apple', 'orange', 'apple', 'pear', 'orange', 'banana'] +>>> for f in sorted(set(basket)): +... print f +... +apple +banana +orange +pear +\end{verbatim} + +\section{More on Conditions \label{conditions}} + +The conditions used in \code{while} and \code{if} statements can +contain any operators, not just comparisons. + +The comparison operators \code{in} and \code{not in} check whether a value +occurs (does not occur) in a sequence. The operators \code{is} and +\code{is not} compare whether two objects are really the same object; this +only matters for mutable objects like lists. All comparison operators +have the same priority, which is lower than that of all numerical +operators. + +Comparisons can be chained. For example, \code{a < b == c} tests +whether \code{a} is less than \code{b} and moreover \code{b} equals +\code{c}. + +Comparisons may be combined using the Boolean operators \code{and} and +\code{or}, and the outcome of a comparison (or of any other Boolean +expression) may be negated with \code{not}. These have lower +priorities than comparison operators; between them, \code{not} has +the highest priority and \code{or} the lowest, so that +\code{A and not B or C} is equivalent to \code{(A and (not B)) or C}. +As always, parentheses can be used to express the desired composition. + +The Boolean operators \code{and} and \code{or} are so-called +\emph{short-circuit} operators: their arguments are evaluated from +left to right, and evaluation stops as soon as the outcome is +determined. For example, if \code{A} and \code{C} are true but +\code{B} is false, \code{A and B and C} does not evaluate the +expression \code{C}. When used as a general value and not as a +Boolean, the return value of a short-circuit operator is the last +evaluated argument. + +It is possible to assign the result of a comparison or other Boolean +expression to a variable. For example, + +\begin{verbatim} +>>> string1, string2, string3 = '', 'Trondheim', 'Hammer Dance' +>>> non_null = string1 or string2 or string3 +>>> non_null +'Trondheim' +\end{verbatim} + +Note that in Python, unlike C, assignment cannot occur inside expressions. +C programmers may grumble about this, but it avoids a common class of +problems encountered in C programs: typing \code{=} in an expression when +\code{==} was intended. + + +\section{Comparing Sequences and Other Types \label{comparing}} + +Sequence objects may be compared to other objects with the same +sequence type. The comparison uses \emph{lexicographical} ordering: +first the first two items are compared, and if they differ this +determines the outcome of the comparison; if they are equal, the next +two items are compared, and so on, until either sequence is exhausted. +If two items to be compared are themselves sequences of the same type, +the lexicographical comparison is carried out recursively. If all +items of two sequences compare equal, the sequences are considered +equal. If one sequence is an initial sub-sequence of the other, the +shorter sequence is the smaller (lesser) one. Lexicographical +ordering for strings uses the \ASCII{} ordering for individual +characters. Some examples of comparisons between sequences of the +same type: + +\begin{verbatim} +(1, 2, 3) < (1, 2, 4) +[1, 2, 3] < [1, 2, 4] +'ABC' < 'C' < 'Pascal' < 'Python' +(1, 2, 3, 4) < (1, 2, 4) +(1, 2) < (1, 2, -1) +(1, 2, 3) == (1.0, 2.0, 3.0) +(1, 2, ('aa', 'ab')) < (1, 2, ('abc', 'a'), 4) +\end{verbatim} + +Note that comparing objects of different types is legal. The outcome +is deterministic but arbitrary: the types are ordered by their name. +Thus, a list is always smaller than a string, a string is always +smaller than a tuple, etc. \footnote{ + The rules for comparing objects of different types should + not be relied upon; they may change in a future version of + the language. +} Mixed numeric types are compared according to their numeric value, so +0 equals 0.0, etc. + + +\chapter{Modules \label{modules}} + +If you quit from the Python interpreter and enter it again, the +definitions you have made (functions and variables) are lost. +Therefore, if you want to write a somewhat longer program, you are +better off using a text editor to prepare the input for the interpreter +and running it with that file as input instead. This is known as creating a +\emph{script}. As your program gets longer, you may want to split it +into several files for easier maintenance. You may also want to use a +handy function that you've written in several programs without copying +its definition into each program. + +To support this, Python has a way to put definitions in a file and use +them in a script or in an interactive instance of the interpreter. +Such a file is called a \emph{module}; definitions from a module can be +\emph{imported} into other modules or into the \emph{main} module (the +collection of variables that you have access to in a script +executed at the top level +and in calculator mode). + +A module is a file containing Python definitions and statements. The +file name is the module name with the suffix \file{.py} appended. Within +a module, the module's name (as a string) is available as the value of +the global variable \code{__name__}. For instance, use your favorite text +editor to create a file called \file{fibo.py} in the current directory +with the following contents: + +\begin{verbatim} +# Fibonacci numbers module + +def fib(n): # write Fibonacci series up to n + a, b = 0, 1 + while b < n: + print b, + a, b = b, a+b + +def fib2(n): # return Fibonacci series up to n + result = [] + a, b = 0, 1 + while b < n: + result.append(b) + a, b = b, a+b + return result +\end{verbatim} + +Now enter the Python interpreter and import this module with the +following command: + +\begin{verbatim} +>>> import fibo +\end{verbatim} + +This does not enter the names of the functions defined in \code{fibo} +directly in the current symbol table; it only enters the module name +\code{fibo} there. +Using the module name you can access the functions: + +\begin{verbatim} +>>> fibo.fib(1000) +1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987 +>>> fibo.fib2(100) +[1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89] +>>> fibo.__name__ +'fibo' +\end{verbatim} + +If you intend to use a function often you can assign it to a local name: + +\begin{verbatim} +>>> fib = fibo.fib +>>> fib(500) +1 1 2 3 5 8 13 21 34 55 89 144 233 377 +\end{verbatim} + + +\section{More on Modules \label{moreModules}} + +A module can contain executable statements as well as function +definitions. +These statements are intended to initialize the module. +They are executed only the +\emph{first} time the module is imported somewhere.\footnote{ + In fact function definitions are also `statements' that are + `executed'; the execution enters the function name in the + module's global symbol table. +} + +Each module has its own private symbol table, which is used as the +global symbol table by all functions defined in the module. +Thus, the author of a module can use global variables in the module +without worrying about accidental clashes with a user's global +variables. +On the other hand, if you know what you are doing you can touch a +module's global variables with the same notation used to refer to its +functions, +\code{modname.itemname}. + +Modules can import other modules. It is customary but not required to +place all \keyword{import} statements at the beginning of a module (or +script, for that matter). The imported module names are placed in the +importing module's global symbol table. + +There is a variant of the \keyword{import} statement that imports +names from a module directly into the importing module's symbol +table. For example: + +\begin{verbatim} +>>> from fibo import fib, fib2 +>>> fib(500) +1 1 2 3 5 8 13 21 34 55 89 144 233 377 +\end{verbatim} + +This does not introduce the module name from which the imports are taken +in the local symbol table (so in the example, \code{fibo} is not +defined). + +There is even a variant to import all names that a module defines: + +\begin{verbatim} +>>> from fibo import * +>>> fib(500) +1 1 2 3 5 8 13 21 34 55 89 144 233 377 +\end{verbatim} + +This imports all names except those beginning with an underscore +(\code{_}). + + +\subsection{The Module Search Path \label{searchPath}} + +\indexiii{module}{search}{path} +When a module named \module{spam} is imported, the interpreter searches +for a file named \file{spam.py} in the current directory, +and then in the list of directories specified by +the environment variable \envvar{PYTHONPATH}. This has the same syntax as +the shell variable \envvar{PATH}, that is, a list of +directory names. When \envvar{PYTHONPATH} is not set, or when the file +is not found there, the search continues in an installation-dependent +default path; on \UNIX, this is usually \file{.:/usr/local/lib/python}. + +Actually, modules are searched in the list of directories given by the +variable \code{sys.path} which is initialized from the directory +containing the input script (or the current directory), +\envvar{PYTHONPATH} and the installation-dependent default. This allows +Python programs that know what they're doing to modify or replace the +module search path. Note that because the directory containing the +script being run is on the search path, it is important that the +script not have the same name as a standard module, or Python will +attempt to load the script as a module when that module is imported. +This will generally be an error. See section~\ref{standardModules}, +``Standard Modules,'' for more information. + + +\subsection{``Compiled'' Python files} + +As an important speed-up of the start-up time for short programs that +use a lot of standard modules, if a file called \file{spam.pyc} exists +in the directory where \file{spam.py} is found, this is assumed to +contain an already-``byte-compiled'' version of the module \module{spam}. +The modification time of the version of \file{spam.py} used to create +\file{spam.pyc} is recorded in \file{spam.pyc}, and the +\file{.pyc} file is ignored if these don't match. + +Normally, you don't need to do anything to create the +\file{spam.pyc} file. Whenever \file{spam.py} is successfully +compiled, an attempt is made to write the compiled version to +\file{spam.pyc}. It is not an error if this attempt fails; if for any +reason the file is not written completely, the resulting +\file{spam.pyc} file will be recognized as invalid and thus ignored +later. The contents of the \file{spam.pyc} file are platform +independent, so a Python module directory can be shared by machines of +different architectures. + +Some tips for experts: + +\begin{itemize} + +\item +When the Python interpreter is invoked with the \programopt{-O} flag, +optimized code is generated and stored in \file{.pyo} files. The +optimizer currently doesn't help much; it only removes +\keyword{assert} statements. When \programopt{-O} is used, \emph{all} +bytecode is optimized; \code{.pyc} files are ignored and \code{.py} +files are compiled to optimized bytecode. + +\item +Passing two \programopt{-O} flags to the Python interpreter +(\programopt{-OO}) will cause the bytecode compiler to perform +optimizations that could in some rare cases result in malfunctioning +programs. Currently only \code{__doc__} strings are removed from the +bytecode, resulting in more compact \file{.pyo} files. Since some +programs may rely on having these available, you should only use this +option if you know what you're doing. + +\item +A program doesn't run any faster when it is read from a \file{.pyc} or +\file{.pyo} file than when it is read from a \file{.py} file; the only +thing that's faster about \file{.pyc} or \file{.pyo} files is the +speed with which they are loaded. + +\item +When a script is run by giving its name on the command line, the +bytecode for the script is never written to a \file{.pyc} or +\file{.pyo} file. Thus, the startup time of a script may be reduced +by moving most of its code to a module and having a small bootstrap +script that imports that module. It is also possible to name a +\file{.pyc} or \file{.pyo} file directly on the command line. + +\item +It is possible to have a file called \file{spam.pyc} (or +\file{spam.pyo} when \programopt{-O} is used) without a file +\file{spam.py} for the same module. This can be used to distribute a +library of Python code in a form that is moderately hard to reverse +engineer. + +\item +The module \ulink{\module{compileall}}{../lib/module-compileall.html}% +{} \refstmodindex{compileall} can create \file{.pyc} files (or +\file{.pyo} files when \programopt{-O} is used) for all modules in a +directory. + +\end{itemize} + + +\section{Standard Modules \label{standardModules}} + +Python comes with a library of standard modules, described in a separate +document, the \citetitle[../lib/lib.html]{Python Library Reference} +(``Library Reference'' hereafter). Some modules are built into the +interpreter; these provide access to operations that are not part of +the core of the language but are nevertheless built in, either for +efficiency or to provide access to operating system primitives such as +system calls. The set of such modules is a configuration option which +also depends on the underlying platform For example, +the \module{amoeba} module is only provided on systems that somehow +support Amoeba primitives. One particular module deserves some +attention: \ulink{\module{sys}}{../lib/module-sys.html}% +\refstmodindex{sys}, which is built into every +Python interpreter. The variables \code{sys.ps1} and +\code{sys.ps2} define the strings used as primary and secondary +prompts: + +\begin{verbatim} +>>> import sys +>>> sys.ps1 +'>>> ' +>>> sys.ps2 +'... ' +>>> sys.ps1 = 'C> ' +C> print 'Yuck!' +Yuck! +C> + +\end{verbatim} + +These two variables are only defined if the interpreter is in +interactive mode. + +The variable \code{sys.path} is a list of strings that determines the +interpreter's search path for modules. It is initialized to a default +path taken from the environment variable \envvar{PYTHONPATH}, or from +a built-in default if \envvar{PYTHONPATH} is not set. You can modify +it using standard list operations: + +\begin{verbatim} +>>> import sys +>>> sys.path.append('/ufs/guido/lib/python') +\end{verbatim} + +\section{The \function{dir()} Function \label{dir}} + +The built-in function \function{dir()} is used to find out which names +a module defines. It returns a sorted list of strings: + +\begin{verbatim} +>>> import fibo, sys +>>> dir(fibo) +['__name__', 'fib', 'fib2'] +>>> dir(sys) +['__displayhook__', '__doc__', '__excepthook__', '__name__', '__stderr__', + '__stdin__', '__stdout__', '_getframe', 'api_version', 'argv', + 'builtin_module_names', 'byteorder', 'callstats', 'copyright', + 'displayhook', 'exc_clear', 'exc_info', 'exc_type', 'excepthook', + 'exec_prefix', 'executable', 'exit', 'getdefaultencoding', 'getdlopenflags', + 'getrecursionlimit', 'getrefcount', 'hexversion', 'maxint', 'maxunicode', + 'meta_path', 'modules', 'path', 'path_hooks', 'path_importer_cache', + 'platform', 'prefix', 'ps1', 'ps2', 'setcheckinterval', 'setdlopenflags', + 'setprofile', 'setrecursionlimit', 'settrace', 'stderr', 'stdin', 'stdout', + 'version', 'version_info', 'warnoptions'] +\end{verbatim} + +Without arguments, \function{dir()} lists the names you have defined +currently: + +\begin{verbatim} +>>> a = [1, 2, 3, 4, 5] +>>> import fibo +>>> fib = fibo.fib +>>> dir() +['__builtins__', '__doc__', '__file__', '__name__', 'a', 'fib', 'fibo', 'sys'] +\end{verbatim} + +Note that it lists all types of names: variables, modules, functions, etc. + +\function{dir()} does not list the names of built-in functions and +variables. If you want a list of those, they are defined in the +standard module \module{__builtin__}\refbimodindex{__builtin__}: + +\begin{verbatim} +>>> import __builtin__ +>>> dir(__builtin__) +['ArithmeticError', 'AssertionError', 'AttributeError', 'DeprecationWarning', + 'EOFError', 'Ellipsis', 'EnvironmentError', 'Exception', 'False', + 'FloatingPointError', 'FutureWarning', 'IOError', 'ImportError', + 'IndentationError', 'IndexError', 'KeyError', 'KeyboardInterrupt', + 'LookupError', 'MemoryError', 'NameError', 'None', 'NotImplemented', + 'NotImplementedError', 'OSError', 'OverflowError', + 'PendingDeprecationWarning', 'ReferenceError', 'RuntimeError', + 'RuntimeWarning', 'StandardError', 'StopIteration', 'SyntaxError', + 'SyntaxWarning', 'SystemError', 'SystemExit', 'TabError', 'True', + 'TypeError', 'UnboundLocalError', 'UnicodeDecodeError', + 'UnicodeEncodeError', 'UnicodeError', 'UnicodeTranslateError', + 'UserWarning', 'ValueError', 'Warning', 'WindowsError', + 'ZeroDivisionError', '_', '__debug__', '__doc__', '__import__', + '__name__', 'abs', 'apply', 'basestring', 'bool', 'buffer', + 'callable', 'chr', 'classmethod', 'cmp', 'coerce', 'compile', + 'complex', 'copyright', 'credits', 'delattr', 'dict', 'dir', 'divmod', + 'enumerate', 'eval', 'execfile', 'exit', 'file', 'filter', 'float', + 'frozenset', 'getattr', 'globals', 'hasattr', 'hash', 'help', 'hex', + 'id', 'input', 'int', 'intern', 'isinstance', 'issubclass', 'iter', + 'len', 'license', 'list', 'locals', 'long', 'map', 'max', 'min', + 'object', 'oct', 'open', 'ord', 'pow', 'property', 'quit', 'range', + 'raw_input', 'reduce', 'reload', 'repr', 'reversed', 'round', 'set', + 'setattr', 'slice', 'sorted', 'staticmethod', 'str', 'sum', 'super', + 'tuple', 'type', 'unichr', 'unicode', 'vars', 'xrange', 'zip'] +\end{verbatim} + + +\section{Packages \label{packages}} + +Packages are a way of structuring Python's module namespace +by using ``dotted module names''. For example, the module name +\module{A.B} designates a submodule named \samp{B} in a package named +\samp{A}. Just like the use of modules saves the authors of different +modules from having to worry about each other's global variable names, +the use of dotted module names saves the authors of multi-module +packages like NumPy or the Python Imaging Library from having to worry +about each other's module names. + +Suppose you want to design a collection of modules (a ``package'') for +the uniform handling of sound files and sound data. There are many +different sound file formats (usually recognized by their extension, +for example: \file{.wav}, \file{.aiff}, \file{.au}), so you may need +to create and maintain a growing collection of modules for the +conversion between the various file formats. There are also many +different operations you might want to perform on sound data (such as +mixing, adding echo, applying an equalizer function, creating an +artificial stereo effect), so in addition you will be writing a +never-ending stream of modules to perform these operations. Here's a +possible structure for your package (expressed in terms of a +hierarchical filesystem): + +\begin{verbatim} +Sound/ Top-level package + __init__.py Initialize the sound package + Formats/ Subpackage for file format conversions + __init__.py + wavread.py + wavwrite.py + aiffread.py + aiffwrite.py + auread.py + auwrite.py + ... + Effects/ Subpackage for sound effects + __init__.py + echo.py + surround.py + reverse.py + ... + Filters/ Subpackage for filters + __init__.py + equalizer.py + vocoder.py + karaoke.py + ... +\end{verbatim} + +When importing the package, Python searches through the directories +on \code{sys.path} looking for the package subdirectory. + +The \file{__init__.py} files are required to make Python treat the +directories as containing packages; this is done to prevent +directories with a common name, such as \samp{string}, from +unintentionally hiding valid modules that occur later on the module +search path. In the simplest case, \file{__init__.py} can just be an +empty file, but it can also execute initialization code for the +package or set the \code{__all__} variable, described later. + +Users of the package can import individual modules from the +package, for example: + +\begin{verbatim} +import Sound.Effects.echo +\end{verbatim} + +This loads the submodule \module{Sound.Effects.echo}. It must be referenced +with its full name. + +\begin{verbatim} +Sound.Effects.echo.echofilter(input, output, delay=0.7, atten=4) +\end{verbatim} + +An alternative way of importing the submodule is: + +\begin{verbatim} +from Sound.Effects import echo +\end{verbatim} + +This also loads the submodule \module{echo}, and makes it available without +its package prefix, so it can be used as follows: + +\begin{verbatim} +echo.echofilter(input, output, delay=0.7, atten=4) +\end{verbatim} + +Yet another variation is to import the desired function or variable directly: + +\begin{verbatim} +from Sound.Effects.echo import echofilter +\end{verbatim} + +Again, this loads the submodule \module{echo}, but this makes its function +\function{echofilter()} directly available: + +\begin{verbatim} +echofilter(input, output, delay=0.7, atten=4) +\end{verbatim} + +Note that when using \code{from \var{package} import \var{item}}, the +item can be either a submodule (or subpackage) of the package, or some +other name defined in the package, like a function, class or +variable. The \code{import} statement first tests whether the item is +defined in the package; if not, it assumes it is a module and attempts +to load it. If it fails to find it, an +\exception{ImportError} exception is raised. + +Contrarily, when using syntax like \code{import +\var{item.subitem.subsubitem}}, each item except for the last must be +a package; the last item can be a module or a package but can't be a +class or function or variable defined in the previous item. + +\subsection{Importing * From a Package \label{pkg-import-star}} +%The \code{__all__} Attribute + +\ttindex{__all__} +Now what happens when the user writes \code{from Sound.Effects import +*}? Ideally, one would hope that this somehow goes out to the +filesystem, finds which submodules are present in the package, and +imports them all. Unfortunately, this operation does not work very +well on Windows platforms, where the filesystem does not +always have accurate information about the case of a filename! On +these platforms, there is no guaranteed way to know whether a file +\file{ECHO.PY} should be imported as a module \module{echo}, +\module{Echo} or \module{ECHO}. (For example, Windows 95 has the +annoying practice of showing all file names with a capitalized first +letter.) The DOS 8+3 filename restriction adds another interesting +problem for long module names. + +The only solution is for the package author to provide an explicit +index of the package. The import statement uses the following +convention: if a package's \file{__init__.py} code defines a list +named \code{__all__}, it is taken to be the list of module names that +should be imported when \code{from \var{package} import *} is +encountered. It is up to the package author to keep this list +up-to-date when a new version of the package is released. Package +authors may also decide not to support it, if they don't see a use for +importing * from their package. For example, the file +\file{Sounds/Effects/__init__.py} could contain the following code: + +\begin{verbatim} +__all__ = ["echo", "surround", "reverse"] +\end{verbatim} + +This would mean that \code{from Sound.Effects import *} would +import the three named submodules of the \module{Sound} package. + +If \code{__all__} is not defined, the statement \code{from Sound.Effects +import *} does \emph{not} import all submodules from the package +\module{Sound.Effects} into the current namespace; it only ensures that the +package \module{Sound.Effects} has been imported (possibly running any +initialization code in \file{__init__.py}) and then imports whatever names are +defined in the package. This includes any names defined (and +submodules explicitly loaded) by \file{__init__.py}. It also includes any +submodules of the package that were explicitly loaded by previous +import statements. Consider this code: + +\begin{verbatim} +import Sound.Effects.echo +import Sound.Effects.surround +from Sound.Effects import * +\end{verbatim} + +In this example, the echo and surround modules are imported in the +current namespace because they are defined in the +\module{Sound.Effects} package when the \code{from...import} statement +is executed. (This also works when \code{__all__} is defined.) + +Note that in general the practice of importing \code{*} from a module or +package is frowned upon, since it often causes poorly readable code. +However, it is okay to use it to save typing in interactive sessions, +and certain modules are designed to export only names that follow +certain patterns. + +Remember, there is nothing wrong with using \code{from Package +import specific_submodule}! In fact, this is the +recommended notation unless the importing module needs to use +submodules with the same name from different packages. + + +\subsection{Intra-package References} + +The submodules often need to refer to each other. For example, the +\module{surround} module might use the \module{echo} module. In fact, +such references are so common that the \keyword{import} statement +first looks in the containing package before looking in the standard +module search path. Thus, the \module{surround} module can simply use +\code{import echo} or \code{from echo import echofilter}. If the +imported module is not found in the current package (the package of +which the current module is a submodule), the \keyword{import} +statement looks for a top-level module with the given name. + +When packages are structured into subpackages (as with the +\module{Sound} package in the example), there's no shortcut to refer +to submodules of sibling packages - the full name of the subpackage +must be used. For example, if the module +\module{Sound.Filters.vocoder} needs to use the \module{echo} module +in the \module{Sound.Effects} package, it can use \code{from +Sound.Effects import echo}. + +Starting with Python 2.5, in addition to the implicit relative imports +described above, you can write explicit relative imports with the +\code{from module import name} form of import statement. These explicit +relative imports use leading dots to indicate the current and parent +packages involved in the relative import. From the \module{surround} +module for example, you might use: + +\begin{verbatim} +from . import echo +from .. import Formats +from ..Filters import equalizer +\end{verbatim} + +Note that both explicit and implicit relative imports are based on the +name of the current module. Since the name of the main module is always +\code{"__main__"}, modules intended for use as the main module of a +Python application should always use absolute imports. + +\subsection{Packages in Multiple Directories} + +Packages support one more special attribute, \member{__path__}. This +is initialized to be a list containing the name of the directory +holding the package's \file{__init__.py} before the code in that file +is executed. This variable can be modified; doing so affects future +searches for modules and subpackages contained in the package. + +While this feature is not often needed, it can be used to extend the +set of modules found in a package. + + + +\chapter{Input and Output \label{io}} + +There are several ways to present the output of a program; data can be +printed in a human-readable form, or written to a file for future use. +This chapter will discuss some of the possibilities. + + +\section{Fancier Output Formatting \label{formatting}} + +So far we've encountered two ways of writing values: \emph{expression +statements} and the \keyword{print} statement. (A third way is using +the \method{write()} method of file objects; the standard output file +can be referenced as \code{sys.stdout}. See the Library Reference for +more information on this.) + +Often you'll want more control over the formatting of your output than +simply printing space-separated values. There are two ways to format +your output; the first way is to do all the string handling yourself; +using string slicing and concatenation operations you can create any +layout you can imagine. The standard module +\module{string}\refstmodindex{string} contains some useful operations +for padding strings to a given column width; these will be discussed +shortly. The second way is to use the \code{\%} operator with a +string as the left argument. The \code{\%} operator interprets the +left argument much like a \cfunction{sprintf()}-style format +string to be applied to the right argument, and returns the string +resulting from this formatting operation. + +One question remains, of course: how do you convert values to strings? +Luckily, Python has ways to convert any value to a string: pass it to +the \function{repr()} or \function{str()} functions. Reverse quotes +(\code{``}) are equivalent to \function{repr()}, but they are no +longer used in modern Python code and will likely not be in future +versions of the language. + +The \function{str()} function is meant to return representations of +values which are fairly human-readable, while \function{repr()} is +meant to generate representations which can be read by the interpreter +(or will force a \exception{SyntaxError} if there is not equivalent +syntax). For objects which don't have a particular representation for +human consumption, \function{str()} will return the same value as +\function{repr()}. Many values, such as numbers or structures like +lists and dictionaries, have the same representation using either +function. Strings and floating point numbers, in particular, have two +distinct representations. + +Some examples: + +\begin{verbatim} +>>> s = 'Hello, world.' +>>> str(s) +'Hello, world.' +>>> repr(s) +"'Hello, world.'" +>>> str(0.1) +'0.1' +>>> repr(0.1) +'0.10000000000000001' +>>> x = 10 * 3.25 +>>> y = 200 * 200 +>>> s = 'The value of x is ' + repr(x) + ', and y is ' + repr(y) + '...' +>>> print s +The value of x is 32.5, and y is 40000... +>>> # The repr() of a string adds string quotes and backslashes: +... hello = 'hello, world\n' +>>> hellos = repr(hello) +>>> print hellos +'hello, world\n' +>>> # The argument to repr() may be any Python object: +... repr((x, y, ('spam', 'eggs'))) +"(32.5, 40000, ('spam', 'eggs'))" +>>> # reverse quotes are convenient in interactive sessions: +... `x, y, ('spam', 'eggs')` +"(32.5, 40000, ('spam', 'eggs'))" +\end{verbatim} + +Here are two ways to write a table of squares and cubes: + +\begin{verbatim} +>>> for x in range(1, 11): +... print repr(x).rjust(2), repr(x*x).rjust(3), +... # Note trailing comma on previous line +... print repr(x*x*x).rjust(4) +... + 1 1 1 + 2 4 8 + 3 9 27 + 4 16 64 + 5 25 125 + 6 36 216 + 7 49 343 + 8 64 512 + 9 81 729 +10 100 1000 + +>>> for x in range(1,11): +... print '%2d %3d %4d' % (x, x*x, x*x*x) +... + 1 1 1 + 2 4 8 + 3 9 27 + 4 16 64 + 5 25 125 + 6 36 216 + 7 49 343 + 8 64 512 + 9 81 729 +10 100 1000 +\end{verbatim} + +(Note that in the first example, one space between each column was +added by the way \keyword{print} works: it always adds spaces between +its arguments.) + +This example demonstrates the \method{rjust()} method of string objects, +which right-justifies a string in a field of a given width by padding +it with spaces on the left. There are similar methods +\method{ljust()} and \method{center()}. These +methods do not write anything, they just return a new string. If +the input string is too long, they don't truncate it, but return it +unchanged; this will mess up your column lay-out but that's usually +better than the alternative, which would be lying about a value. (If +you really want truncation you can always add a slice operation, as in +\samp{x.ljust(n)[:n]}.) + +There is another method, \method{zfill()}, which pads a +numeric string on the left with zeros. It understands about plus and +minus signs: + +\begin{verbatim} +>>> '12'.zfill(5) +'00012' +>>> '-3.14'.zfill(7) +'-003.14' +>>> '3.14159265359'.zfill(5) +'3.14159265359' +\end{verbatim} + +Using the \code{\%} operator looks like this: + +\begin{verbatim} +>>> import math +>>> print 'The value of PI is approximately %5.3f.' % math.pi +The value of PI is approximately 3.142. +\end{verbatim} + +If there is more than one format in the string, you need to pass a +tuple as right operand, as in this example: + +\begin{verbatim} +>>> table = {'Sjoerd': 4127, 'Jack': 4098, 'Dcab': 7678} +>>> for name, phone in table.items(): +... print '%-10s ==> %10d' % (name, phone) +... +Jack ==> 4098 +Dcab ==> 7678 +Sjoerd ==> 4127 +\end{verbatim} + +Most formats work exactly as in C and require that you pass the proper +type; however, if you don't you get an exception, not a core dump. +The \code{\%s} format is more relaxed: if the corresponding argument is +not a string object, it is converted to string using the +\function{str()} built-in function. Using \code{*} to pass the width +or precision in as a separate (integer) argument is supported. The +C formats \code{\%n} and \code{\%p} are not supported. + +If you have a really long format string that you don't want to split +up, it would be nice if you could reference the variables to be +formatted by name instead of by position. This can be done by using +form \code{\%(name)format}, as shown here: + +\begin{verbatim} +>>> table = {'Sjoerd': 4127, 'Jack': 4098, 'Dcab': 8637678} +>>> print 'Jack: %(Jack)d; Sjoerd: %(Sjoerd)d; Dcab: %(Dcab)d' % table +Jack: 4098; Sjoerd: 4127; Dcab: 8637678 +\end{verbatim} + +This is particularly useful in combination with the new built-in +\function{vars()} function, which returns a dictionary containing all +local variables. + +\section{Reading and Writing Files \label{files}} + +% Opening files +\function{open()}\bifuncindex{open} returns a file +object\obindex{file}, and is most commonly used with two arguments: +\samp{open(\var{filename}, \var{mode})}. + +\begin{verbatim} +>>> f=open('/tmp/workfile', 'w') +>>> print f +<open file '/tmp/workfile', mode 'w' at 80a0960> +\end{verbatim} + +The first argument is a string containing the filename. The second +argument is another string containing a few characters describing the +way in which the file will be used. \var{mode} can be \code{'r'} when +the file will only be read, \code{'w'} for only writing (an existing +file with the same name will be erased), and \code{'a'} opens the file +for appending; any data written to the file is automatically added to +the end. \code{'r+'} opens the file for both reading and writing. +The \var{mode} argument is optional; \code{'r'} will be assumed if +it's omitted. + +On Windows and the Macintosh, \code{'b'} appended to the +mode opens the file in binary mode, so there are also modes like +\code{'rb'}, \code{'wb'}, and \code{'r+b'}. Windows makes a +distinction between text and binary files; the end-of-line characters +in text files are automatically altered slightly when data is read or +written. This behind-the-scenes modification to file data is fine for +\ASCII{} text files, but it'll corrupt binary data like that in \file{JPEG} or +\file{EXE} files. Be very careful to use binary mode when reading and +writing such files. + +\subsection{Methods of File Objects \label{fileMethods}} + +The rest of the examples in this section will assume that a file +object called \code{f} has already been created. + +To read a file's contents, call \code{f.read(\var{size})}, which reads +some quantity of data and returns it as a string. \var{size} is an +optional numeric argument. When \var{size} is omitted or negative, +the entire contents of the file will be read and returned; it's your +problem if the file is twice as large as your machine's memory. +Otherwise, at most \var{size} bytes are read and returned. If the end +of the file has been reached, \code{f.read()} will return an empty +string (\code {""}). +\begin{verbatim} +>>> f.read() +'This is the entire file.\n' +>>> f.read() +'' +\end{verbatim} + +\code{f.readline()} reads a single line from the file; a newline +character (\code{\e n}) is left at the end of the string, and is only +omitted on the last line of the file if the file doesn't end in a +newline. This makes the return value unambiguous; if +\code{f.readline()} returns an empty string, the end of the file has +been reached, while a blank line is represented by \code{'\e n'}, a +string containing only a single newline. + +\begin{verbatim} +>>> f.readline() +'This is the first line of the file.\n' +>>> f.readline() +'Second line of the file\n' +>>> f.readline() +'' +\end{verbatim} + +\code{f.readlines()} returns a list containing all the lines of data +in the file. If given an optional parameter \var{sizehint}, it reads +that many bytes from the file and enough more to complete a line, and +returns the lines from that. This is often used to allow efficient +reading of a large file by lines, but without having to load the +entire file in memory. Only complete lines will be returned. + +\begin{verbatim} +>>> f.readlines() +['This is the first line of the file.\n', 'Second line of the file\n'] +\end{verbatim} + +An alternate approach to reading lines is to loop over the file object. +This is memory efficient, fast, and leads to simpler code: + +\begin{verbatim} +>>> for line in f: + print line, + +This is the first line of the file. +Second line of the file +\end{verbatim} + +The alternative approach is simpler but does not provide as fine-grained +control. Since the two approaches manage line buffering differently, +they should not be mixed. + +\code{f.write(\var{string})} writes the contents of \var{string} to +the file, returning \code{None}. + +\begin{verbatim} +>>> f.write('This is a test\n') +\end{verbatim} + +To write something other than a string, it needs to be converted to a +string first: + +\begin{verbatim} +>>> value = ('the answer', 42) +>>> s = str(value) +>>> f.write(s) +\end{verbatim} + +\code{f.tell()} returns an integer giving the file object's current +position in the file, measured in bytes from the beginning of the +file. To change the file object's position, use +\samp{f.seek(\var{offset}, \var{from_what})}. The position is +computed from adding \var{offset} to a reference point; the reference +point is selected by the \var{from_what} argument. A +\var{from_what} value of 0 measures from the beginning of the file, 1 +uses the current file position, and 2 uses the end of the file as the +reference point. \var{from_what} can be omitted and defaults to 0, +using the beginning of the file as the reference point. + +\begin{verbatim} +>>> f = open('/tmp/workfile', 'r+') +>>> f.write('0123456789abcdef') +>>> f.seek(5) # Go to the 6th byte in the file +>>> f.read(1) +'5' +>>> f.seek(-3, 2) # Go to the 3rd byte before the end +>>> f.read(1) +'d' +\end{verbatim} + +When you're done with a file, call \code{f.close()} to close it and +free up any system resources taken up by the open file. After calling +\code{f.close()}, attempts to use the file object will automatically fail. + +\begin{verbatim} +>>> f.close() +>>> f.read() +Traceback (most recent call last): + File "<stdin>", line 1, in ? +ValueError: I/O operation on closed file +\end{verbatim} + +File objects have some additional methods, such as +\method{isatty()} and \method{truncate()} which are less frequently +used; consult the Library Reference for a complete guide to file +objects. + +\subsection{The \module{pickle} Module \label{pickle}} +\refstmodindex{pickle} + +Strings can easily be written to and read from a file. Numbers take a +bit more effort, since the \method{read()} method only returns +strings, which will have to be passed to a function like +\function{int()}, which takes a string like \code{'123'} and +returns its numeric value 123. However, when you want to save more +complex data types like lists, dictionaries, or class instances, +things get a lot more complicated. + +Rather than have users be constantly writing and debugging code to +save complicated data types, Python provides a standard module called +\ulink{\module{pickle}}{../lib/module-pickle.html}. This is an +amazing module that can take almost +any Python object (even some forms of Python code!), and convert it to +a string representation; this process is called \dfn{pickling}. +Reconstructing the object from the string representation is called +\dfn{unpickling}. Between pickling and unpickling, the string +representing the object may have been stored in a file or data, or +sent over a network connection to some distant machine. + +If you have an object \code{x}, and a file object \code{f} that's been +opened for writing, the simplest way to pickle the object takes only +one line of code: + +\begin{verbatim} +pickle.dump(x, f) +\end{verbatim} + +To unpickle the object again, if \code{f} is a file object which has +been opened for reading: + +\begin{verbatim} +x = pickle.load(f) +\end{verbatim} + +(There are other variants of this, used when pickling many objects or +when you don't want to write the pickled data to a file; consult the +complete documentation for +\ulink{\module{pickle}}{../lib/module-pickle.html} in the +\citetitle[../lib/]{Python Library Reference}.) + +\ulink{\module{pickle}}{../lib/module-pickle.html} is the standard way +to make Python objects which can be stored and reused by other +programs or by a future invocation of the same program; the technical +term for this is a \dfn{persistent} object. Because +\ulink{\module{pickle}}{../lib/module-pickle.html} is so widely used, +many authors who write Python extensions take care to ensure that new +data types such as matrices can be properly pickled and unpickled. + + + +\chapter{Errors and Exceptions \label{errors}} + +Until now error messages haven't been more than mentioned, but if you +have tried out the examples you have probably seen some. There are +(at least) two distinguishable kinds of errors: +\emph{syntax errors} and \emph{exceptions}. + +\section{Syntax Errors \label{syntaxErrors}} + +Syntax errors, also known as parsing errors, are perhaps the most common +kind of complaint you get while you are still learning Python: + +\begin{verbatim} +>>> while True print 'Hello world' + File "<stdin>", line 1, in ? + while True print 'Hello world' + ^ +SyntaxError: invalid syntax +\end{verbatim} + +The parser repeats the offending line and displays a little `arrow' +pointing at the earliest point in the line where the error was +detected. The error is caused by (or at least detected at) the token +\emph{preceding} the arrow: in the example, the error is detected at +the keyword \keyword{print}, since a colon (\character{:}) is missing +before it. File name and line number are printed so you know where to +look in case the input came from a script. + +\section{Exceptions \label{exceptions}} + +Even if a statement or expression is syntactically correct, it may +cause an error when an attempt is made to execute it. +Errors detected during execution are called \emph{exceptions} and are +not unconditionally fatal: you will soon learn how to handle them in +Python programs. Most exceptions are not handled by programs, +however, and result in error messages as shown here: + +\begin{verbatim} +>>> 10 * (1/0) +Traceback (most recent call last): + File "<stdin>", line 1, in ? +ZeroDivisionError: integer division or modulo by zero +>>> 4 + spam*3 +Traceback (most recent call last): + File "<stdin>", line 1, in ? +NameError: name 'spam' is not defined +>>> '2' + 2 +Traceback (most recent call last): + File "<stdin>", line 1, in ? +TypeError: cannot concatenate 'str' and 'int' objects +\end{verbatim} + +The last line of the error message indicates what happened. +Exceptions come in different types, and the type is printed as part of +the message: the types in the example are +\exception{ZeroDivisionError}, \exception{NameError} and +\exception{TypeError}. +The string printed as the exception type is the name of the built-in +exception that occurred. This is true for all built-in +exceptions, but need not be true for user-defined exceptions (although +it is a useful convention). +Standard exception names are built-in identifiers (not reserved +keywords). + +The rest of the line provides detail based on the type of exception +and what caused it. + +The preceding part of the error message shows the context where the +exception happened, in the form of a stack traceback. +In general it contains a stack traceback listing source lines; however, +it will not display lines read from standard input. + +The \citetitle[../lib/module-exceptions.html]{Python Library +Reference} lists the built-in exceptions and their meanings. + + +\section{Handling Exceptions \label{handling}} + +It is possible to write programs that handle selected exceptions. +Look at the following example, which asks the user for input until a +valid integer has been entered, but allows the user to interrupt the +program (using \kbd{Control-C} or whatever the operating system +supports); note that a user-generated interruption is signalled by +raising the \exception{KeyboardInterrupt} exception. + +\begin{verbatim} +>>> while True: +... try: +... x = int(raw_input("Please enter a number: ")) +... break +... except ValueError: +... print "Oops! That was no valid number. Try again..." +... +\end{verbatim} + +The \keyword{try} statement works as follows. + +\begin{itemize} +\item +First, the \emph{try clause} (the statement(s) between the +\keyword{try} and \keyword{except} keywords) is executed. + +\item +If no exception occurs, the \emph{except\ clause} is skipped and +execution of the \keyword{try} statement is finished. + +\item +If an exception occurs during execution of the try clause, the rest of +the clause is skipped. Then if its type matches the exception named +after the \keyword{except} keyword, the except clause is executed, and +then execution continues after the \keyword{try} statement. + +\item +If an exception occurs which does not match the exception named in the +except clause, it is passed on to outer \keyword{try} statements; if +no handler is found, it is an \emph{unhandled exception} and execution +stops with a message as shown above. + +\end{itemize} + +A \keyword{try} statement may have more than one except clause, to +specify handlers for different exceptions. At most one handler will +be executed. Handlers only handle exceptions that occur in the +corresponding try clause, not in other handlers of the same +\keyword{try} statement. An except clause may name multiple exceptions +as a parenthesized tuple, for example: + +\begin{verbatim} +... except (RuntimeError, TypeError, NameError): +... pass +\end{verbatim} + +The last except clause may omit the exception name(s), to serve as a +wildcard. Use this with extreme caution, since it is easy to mask a +real programming error in this way! It can also be used to print an +error message and then re-raise the exception (allowing a caller to +handle the exception as well): + +\begin{verbatim} +import sys + +try: + f = open('myfile.txt') + s = f.readline() + i = int(s.strip()) +except IOError, (errno, strerror): + print "I/O error(%s): %s" % (errno, strerror) +except ValueError: + print "Could not convert data to an integer." +except: + print "Unexpected error:", sys.exc_info()[0] + raise +\end{verbatim} + +The \keyword{try} \ldots\ \keyword{except} statement has an optional +\emph{else clause}, which, when present, must follow all except +clauses. It is useful for code that must be executed if the try +clause does not raise an exception. For example: + +\begin{verbatim} +for arg in sys.argv[1:]: + try: + f = open(arg, 'r') + except IOError: + print 'cannot open', arg + else: + print arg, 'has', len(f.readlines()), 'lines' + f.close() +\end{verbatim} + +The use of the \keyword{else} clause is better than adding additional +code to the \keyword{try} clause because it avoids accidentally +catching an exception that wasn't raised by the code being protected +by the \keyword{try} \ldots\ \keyword{except} statement. + + +When an exception occurs, it may have an associated value, also known as +the exception's \emph{argument}. +The presence and type of the argument depend on the exception type. + +The except clause may specify a variable after the exception name (or tuple). +The variable is bound to an exception instance with the arguments stored +in \code{instance.args}. For convenience, the exception instance +defines \method{__getitem__} and \method{__str__} so the arguments can +be accessed or printed directly without having to reference \code{.args}. + +But use of \code{.args} is discouraged. Instead, the preferred use is to pass +a single argument to an exception (which can be a tuple if multiple arguments +are needed) and have it bound to the \code{message} attribute. One may also +instantiate an exception first before raising it and add any attributes to it +as desired. + +\begin{verbatim} +>>> try: +... raise Exception('spam', 'eggs') +... except Exception, inst: +... print type(inst) # the exception instance +... print inst.args # arguments stored in .args +... print inst # __str__ allows args to printed directly +... x, y = inst # __getitem__ allows args to be unpacked directly +... print 'x =', x +... print 'y =', y +... +<type 'instance'> +('spam', 'eggs') +('spam', 'eggs') +x = spam +y = eggs +\end{verbatim} + +If an exception has an argument, it is printed as the last part +(`detail') of the message for unhandled exceptions. + +Exception handlers don't just handle exceptions if they occur +immediately in the try clause, but also if they occur inside functions +that are called (even indirectly) in the try clause. +For example: + +\begin{verbatim} +>>> def this_fails(): +... x = 1/0 +... +>>> try: +... this_fails() +... except ZeroDivisionError, detail: +... print 'Handling run-time error:', detail +... +Handling run-time error: integer division or modulo by zero +\end{verbatim} + + +\section{Raising Exceptions \label{raising}} + +The \keyword{raise} statement allows the programmer to force a +specified exception to occur. +For example: + +\begin{verbatim} +>>> raise NameError, 'HiThere' +Traceback (most recent call last): + File "<stdin>", line 1, in ? +NameError: HiThere +\end{verbatim} + +The first argument to \keyword{raise} names the exception to be +raised. The optional second argument specifies the exception's +argument. Alternatively, the above could be written as +\code{raise NameError('HiThere')}. Either form works fine, but there +seems to be a growing stylistic preference for the latter. + +If you need to determine whether an exception was raised but don't +intend to handle it, a simpler form of the \keyword{raise} statement +allows you to re-raise the exception: + +\begin{verbatim} +>>> try: +... raise NameError, 'HiThere' +... except NameError: +... print 'An exception flew by!' +... raise +... +An exception flew by! +Traceback (most recent call last): + File "<stdin>", line 2, in ? +NameError: HiThere +\end{verbatim} + + +\section{User-defined Exceptions \label{userExceptions}} + +Programs may name their own exceptions by creating a new exception +class. Exceptions should typically be derived from the +\exception{Exception} class, either directly or indirectly. For +example: + +\begin{verbatim} +>>> class MyError(Exception): +... def __init__(self, value): +... self.value = value +... def __str__(self): +... return repr(self.value) +... +>>> try: +... raise MyError(2*2) +... except MyError, e: +... print 'My exception occurred, value:', e.value +... +My exception occurred, value: 4 +>>> raise MyError, 'oops!' +Traceback (most recent call last): + File "<stdin>", line 1, in ? +__main__.MyError: 'oops!' +\end{verbatim} + +In this example, the default \method{__init__} of \class{Exception} +has been overridden. The new behavior simply creates the \var{value} +attribute. This replaces the default behavior of creating the +\var{args} attribute. + +Exception classes can be defined which do anything any other class can +do, but are usually kept simple, often only offering a number of +attributes that allow information about the error to be extracted by +handlers for the exception. When creating a module that can raise +several distinct errors, a common practice is to create a base class +for exceptions defined by that module, and subclass that to create +specific exception classes for different error conditions: + +\begin{verbatim} +class Error(Exception): + """Base class for exceptions in this module.""" + pass + +class InputError(Error): + """Exception raised for errors in the input. + + Attributes: + expression -- input expression in which the error occurred + message -- explanation of the error + """ + + def __init__(self, expression, message): + self.expression = expression + self.message = message + +class TransitionError(Error): + """Raised when an operation attempts a state transition that's not + allowed. + + Attributes: + previous -- state at beginning of transition + next -- attempted new state + message -- explanation of why the specific transition is not allowed + """ + + def __init__(self, previous, next, message): + self.previous = previous + self.next = next + self.message = message +\end{verbatim} + +Most exceptions are defined with names that end in ``Error,'' similar +to the naming of the standard exceptions. + +Many standard modules define their own exceptions to report errors +that may occur in functions they define. More information on classes +is presented in chapter \ref{classes}, ``Classes.'' + + +\section{Defining Clean-up Actions \label{cleanup}} + +The \keyword{try} statement has another optional clause which is +intended to define clean-up actions that must be executed under all +circumstances. For example: + +\begin{verbatim} +>>> try: +... raise KeyboardInterrupt +... finally: +... print 'Goodbye, world!' +... +Goodbye, world! +Traceback (most recent call last): + File "<stdin>", line 2, in ? +KeyboardInterrupt +\end{verbatim} + +A \emph{finally clause} is always executed before leaving the +\keyword{try} statement, whether an exception has occurred or not. +When an exception has occurred in the \keyword{try} clause and has not +been handled by an \keyword{except} clause (or it has occurred in a +\keyword{except} or \keyword{else} clause), it is re-raised after the +\keyword{finally} clause has been executed. The \keyword{finally} clause +is also executed ``on the way out'' when any other clause of the +\keyword{try} statement is left via a \keyword{break}, \keyword{continue} +or \keyword{return} statement. A more complicated example: + +\begin{verbatim} +>>> def divide(x, y): +... try: +... result = x / y +... except ZeroDivisionError: +... print "division by zero!" +... else: +... print "result is", result +... finally: +... print "executing finally clause" +... +>>> divide(2, 1) +result is 2 +executing finally clause +>>> divide(2, 0) +division by zero! +executing finally clause +>>> divide("2", "1") +executing finally clause +Traceback (most recent call last): + File "<stdin>", line 1, in ? + File "<stdin>", line 3, in divide +TypeError: unsupported operand type(s) for /: 'str' and 'str' +\end{verbatim} + +As you can see, the \keyword{finally} clause is executed in any +event. The \exception{TypeError} raised by dividing two strings +is not handled by the \keyword{except} clause and therefore +re-raised after the \keyword{finally} clauses has been executed. + +In real world applications, the \keyword{finally} clause is useful +for releasing external resources (such as files or network connections), +regardless of whether the use of the resource was successful. + + +\section{Predefined Clean-up Actions \label{cleanup-with}} + +Some objects define standard clean-up actions to be undertaken when +the object is no longer needed, regardless of whether or not the +operation using the object succeeded or failed. +Look at the following example, which tries to open a file and print +its contents to the screen. + +\begin{verbatim} +for line in open("myfile.txt"): + print line +\end{verbatim} + +The problem with this code is that it leaves the file open for an +indeterminate amount of time after the code has finished executing. +This is not an issue in simple scripts, but can be a problem for +larger applications. The \keyword{with} statement allows +objects like files to be used in a way that ensures they are +always cleaned up promptly and correctly. + +\begin{verbatim} +with open("myfile.txt") as f: + for line in f: + print line +\end{verbatim} + +After the statement is executed, the file \var{f} is always closed, +even if a problem was encountered while processing the lines. Other +objects which provide predefined clean-up actions will indicate +this in their documentation. + + +\chapter{Classes \label{classes}} + +Python's class mechanism adds classes to the language with a minimum +of new syntax and semantics. It is a mixture of the class mechanisms +found in \Cpp{} and Modula-3. As is true for modules, classes in Python +do not put an absolute barrier between definition and user, but rather +rely on the politeness of the user not to ``break into the +definition.'' The most important features of classes are retained +with full power, however: the class inheritance mechanism allows +multiple base classes, a derived class can override any methods of its +base class or classes, and a method can call the method of a base class with the +same name. Objects can contain an arbitrary amount of private data. + +In \Cpp{} terminology, all class members (including the data members) are +\emph{public}, and all member functions are \emph{virtual}. There are +no special constructors or destructors. As in Modula-3, there are no +shorthands for referencing the object's members from its methods: the +method function is declared with an explicit first argument +representing the object, which is provided implicitly by the call. As +in Smalltalk, classes themselves are objects, albeit in the wider +sense of the word: in Python, all data types are objects. This +provides semantics for importing and renaming. Unlike +\Cpp{} and Modula-3, built-in types can be used as base classes for +extension by the user. Also, like in \Cpp{} but unlike in Modula-3, most +built-in operators with special syntax (arithmetic operators, +subscripting etc.) can be redefined for class instances. + +\section{A Word About Terminology \label{terminology}} + +Lacking universally accepted terminology to talk about classes, I will +make occasional use of Smalltalk and \Cpp{} terms. (I would use Modula-3 +terms, since its object-oriented semantics are closer to those of +Python than \Cpp, but I expect that few readers have heard of it.) + +Objects have individuality, and multiple names (in multiple scopes) +can be bound to the same object. This is known as aliasing in other +languages. This is usually not appreciated on a first glance at +Python, and can be safely ignored when dealing with immutable basic +types (numbers, strings, tuples). However, aliasing has an +(intended!) effect on the semantics of Python code involving mutable +objects such as lists, dictionaries, and most types representing +entities outside the program (files, windows, etc.). This is usually +used to the benefit of the program, since aliases behave like pointers +in some respects. For example, passing an object is cheap since only +a pointer is passed by the implementation; and if a function modifies +an object passed as an argument, the caller will see the change --- this +eliminates the need for two different argument passing mechanisms as in +Pascal. + + +\section{Python Scopes and Name Spaces \label{scopes}} + +Before introducing classes, I first have to tell you something about +Python's scope rules. Class definitions play some neat tricks with +namespaces, and you need to know how scopes and namespaces work to +fully understand what's going on. Incidentally, knowledge about this +subject is useful for any advanced Python programmer. + +Let's begin with some definitions. + +A \emph{namespace} is a mapping from names to objects. Most +namespaces are currently implemented as Python dictionaries, but +that's normally not noticeable in any way (except for performance), +and it may change in the future. Examples of namespaces are: the set +of built-in names (functions such as \function{abs()}, and built-in +exception names); the global names in a module; and the local names in +a function invocation. In a sense the set of attributes of an object +also form a namespace. The important thing to know about namespaces +is that there is absolutely no relation between names in different +namespaces; for instance, two different modules may both define a +function ``maximize'' without confusion --- users of the modules must +prefix it with the module name. + +By the way, I use the word \emph{attribute} for any name following a +dot --- for example, in the expression \code{z.real}, \code{real} is +an attribute of the object \code{z}. Strictly speaking, references to +names in modules are attribute references: in the expression +\code{modname.funcname}, \code{modname} is a module object and +\code{funcname} is an attribute of it. In this case there happens to +be a straightforward mapping between the module's attributes and the +global names defined in the module: they share the same namespace! +\footnote{ + Except for one thing. Module objects have a secret read-only + attribute called \member{__dict__} which returns the dictionary + used to implement the module's namespace; the name + \member{__dict__} is an attribute but not a global name. + Obviously, using this violates the abstraction of namespace + implementation, and should be restricted to things like + post-mortem debuggers. +} + +Attributes may be read-only or writable. In the latter case, +assignment to attributes is possible. Module attributes are writable: +you can write \samp{modname.the_answer = 42}. Writable attributes may +also be deleted with the \keyword{del} statement. For example, +\samp{del modname.the_answer} will remove the attribute +\member{the_answer} from the object named by \code{modname}. + +Name spaces are created at different moments and have different +lifetimes. The namespace containing the built-in names is created +when the Python interpreter starts up, and is never deleted. The +global namespace for a module is created when the module definition +is read in; normally, module namespaces also last until the +interpreter quits. The statements executed by the top-level +invocation of the interpreter, either read from a script file or +interactively, are considered part of a module called +\module{__main__}, so they have their own global namespace. (The +built-in names actually also live in a module; this is called +\module{__builtin__}.) + +The local namespace for a function is created when the function is +called, and deleted when the function returns or raises an exception +that is not handled within the function. (Actually, forgetting would +be a better way to describe what actually happens.) Of course, +recursive invocations each have their own local namespace. + +A \emph{scope} is a textual region of a Python program where a +namespace is directly accessible. ``Directly accessible'' here means +that an unqualified reference to a name attempts to find the name in +the namespace. + +Although scopes are determined statically, they are used dynamically. +At any time during execution, there are at least three nested scopes whose +namespaces are directly accessible: the innermost scope, which is searched +first, contains the local names; the namespaces of any enclosing +functions, which are searched starting with the nearest enclosing scope; +the middle scope, searched next, contains the current module's global names; +and the outermost scope (searched last) is the namespace containing built-in +names. + +If a name is declared global, then all references and assignments go +directly to the middle scope containing the module's global names. +Otherwise, all variables found outside of the innermost scope are read-only +(an attempt to write to such a variable will simply create a \emph{new} +local variable in the innermost scope, leaving the identically named +outer variable unchanged). + +Usually, the local scope references the local names of the (textually) +current function. Outside functions, the local scope references +the same namespace as the global scope: the module's namespace. +Class definitions place yet another namespace in the local scope. + +It is important to realize that scopes are determined textually: the +global scope of a function defined in a module is that module's +namespace, no matter from where or by what alias the function is +called. On the other hand, the actual search for names is done +dynamically, at run time --- however, the language definition is +evolving towards static name resolution, at ``compile'' time, so don't +rely on dynamic name resolution! (In fact, local variables are +already determined statically.) + +A special quirk of Python is that assignments always go into the +innermost scope. Assignments do not copy data --- they just +bind names to objects. The same is true for deletions: the statement +\samp{del x} removes the binding of \code{x} from the namespace +referenced by the local scope. In fact, all operations that introduce +new names use the local scope: in particular, import statements and +function definitions bind the module or function name in the local +scope. (The \keyword{global} statement can be used to indicate that +particular variables live in the global scope.) + + +\section{A First Look at Classes \label{firstClasses}} + +Classes introduce a little bit of new syntax, three new object types, +and some new semantics. + + +\subsection{Class Definition Syntax \label{classDefinition}} + +The simplest form of class definition looks like this: + +\begin{verbatim} +class ClassName: + <statement-1> + . + . + . + <statement-N> +\end{verbatim} + +Class definitions, like function definitions +(\keyword{def} statements) must be executed before they have any +effect. (You could conceivably place a class definition in a branch +of an \keyword{if} statement, or inside a function.) + +In practice, the statements inside a class definition will usually be +function definitions, but other statements are allowed, and sometimes +useful --- we'll come back to this later. The function definitions +inside a class normally have a peculiar form of argument list, +dictated by the calling conventions for methods --- again, this is +explained later. + +When a class definition is entered, a new namespace is created, and +used as the local scope --- thus, all assignments to local variables +go into this new namespace. In particular, function definitions bind +the name of the new function here. + +When a class definition is left normally (via the end), a \emph{class +object} is created. This is basically a wrapper around the contents +of the namespace created by the class definition; we'll learn more +about class objects in the next section. The original local scope +(the one in effect just before the class definition was entered) is +reinstated, and the class object is bound here to the class name given +in the class definition header (\class{ClassName} in the example). + + +\subsection{Class Objects \label{classObjects}} + +Class objects support two kinds of operations: attribute references +and instantiation. + +\emph{Attribute references} use the standard syntax used for all +attribute references in Python: \code{obj.name}. Valid attribute +names are all the names that were in the class's namespace when the +class object was created. So, if the class definition looked like +this: + +\begin{verbatim} +class MyClass: + "A simple example class" + i = 12345 + def f(self): + return 'hello world' +\end{verbatim} + +then \code{MyClass.i} and \code{MyClass.f} are valid attribute +references, returning an integer and a function object, respectively. +Class attributes can also be assigned to, so you can change the value +of \code{MyClass.i} by assignment. \member{__doc__} is also a valid +attribute, returning the docstring belonging to the class: \code{"A +simple example class"}. + +Class \emph{instantiation} uses function notation. Just pretend that +the class object is a parameterless function that returns a new +instance of the class. For example (assuming the above class): + +\begin{verbatim} +x = MyClass() +\end{verbatim} + +creates a new \emph{instance} of the class and assigns this object to +the local variable \code{x}. + +The instantiation operation (``calling'' a class object) creates an +empty object. Many classes like to create objects with instances +customized to a specific initial state. +Therefore a class may define a special method named +\method{__init__()}, like this: + +\begin{verbatim} + def __init__(self): + self.data = [] +\end{verbatim} + +When a class defines an \method{__init__()} method, class +instantiation automatically invokes \method{__init__()} for the +newly-created class instance. So in this example, a new, initialized +instance can be obtained by: + +\begin{verbatim} +x = MyClass() +\end{verbatim} + +Of course, the \method{__init__()} method may have arguments for +greater flexibility. In that case, arguments given to the class +instantiation operator are passed on to \method{__init__()}. For +example, + +\begin{verbatim} +>>> class Complex: +... def __init__(self, realpart, imagpart): +... self.r = realpart +... self.i = imagpart +... +>>> x = Complex(3.0, -4.5) +>>> x.r, x.i +(3.0, -4.5) +\end{verbatim} + + +\subsection{Instance Objects \label{instanceObjects}} + +Now what can we do with instance objects? The only operations +understood by instance objects are attribute references. There are +two kinds of valid attribute names, data attributes and methods. + +\emph{data attributes} correspond to +``instance variables'' in Smalltalk, and to ``data members'' in +\Cpp. Data attributes need not be declared; like local variables, +they spring into existence when they are first assigned to. For +example, if \code{x} is the instance of \class{MyClass} created above, +the following piece of code will print the value \code{16}, without +leaving a trace: + +\begin{verbatim} +x.counter = 1 +while x.counter < 10: + x.counter = x.counter * 2 +print x.counter +del x.counter +\end{verbatim} + +The other kind of instance attribute reference is a \emph{method}. +A method is a function that ``belongs to'' an +object. (In Python, the term method is not unique to class instances: +other object types can have methods as well. For example, list objects have +methods called append, insert, remove, sort, and so on. However, +in the following discussion, we'll use the term method exclusively to mean +methods of class instance objects, unless explicitly stated otherwise.) + +Valid method names of an instance object depend on its class. By +definition, all attributes of a class that are function +objects define corresponding methods of its instances. So in our +example, \code{x.f} is a valid method reference, since +\code{MyClass.f} is a function, but \code{x.i} is not, since +\code{MyClass.i} is not. But \code{x.f} is not the same thing as +\code{MyClass.f} --- it is a \obindex{method}\emph{method object}, not +a function object. + + +\subsection{Method Objects \label{methodObjects}} + +Usually, a method is called right after it is bound: + +\begin{verbatim} +x.f() +\end{verbatim} + +In the \class{MyClass} example, this will return the string \code{'hello world'}. +However, it is not necessary to call a method right away: +\code{x.f} is a method object, and can be stored away and called at a +later time. For example: + +\begin{verbatim} +xf = x.f +while True: + print xf() +\end{verbatim} + +will continue to print \samp{hello world} until the end of time. + +What exactly happens when a method is called? You may have noticed +that \code{x.f()} was called without an argument above, even though +the function definition for \method{f} specified an argument. What +happened to the argument? Surely Python raises an exception when a +function that requires an argument is called without any --- even if +the argument isn't actually used... + +Actually, you may have guessed the answer: the special thing about +methods is that the object is passed as the first argument of the +function. In our example, the call \code{x.f()} is exactly equivalent +to \code{MyClass.f(x)}. In general, calling a method with a list of +\var{n} arguments is equivalent to calling the corresponding function +with an argument list that is created by inserting the method's object +before the first argument. + +If you still don't understand how methods work, a look at the +implementation can perhaps clarify matters. When an instance +attribute is referenced that isn't a data attribute, its class is +searched. If the name denotes a valid class attribute that is a +function object, a method object is created by packing (pointers to) +the instance object and the function object just found together in an +abstract object: this is the method object. When the method object is +called with an argument list, it is unpacked again, a new argument +list is constructed from the instance object and the original argument +list, and the function object is called with this new argument list. + + +\section{Random Remarks \label{remarks}} + +% [These should perhaps be placed more carefully...] + + +Data attributes override method attributes with the same name; to +avoid accidental name conflicts, which may cause hard-to-find bugs in +large programs, it is wise to use some kind of convention that +minimizes the chance of conflicts. Possible conventions include +capitalizing method names, prefixing data attribute names with a small +unique string (perhaps just an underscore), or using verbs for methods +and nouns for data attributes. + + +Data attributes may be referenced by methods as well as by ordinary +users (``clients'') of an object. In other words, classes are not +usable to implement pure abstract data types. In fact, nothing in +Python makes it possible to enforce data hiding --- it is all based +upon convention. (On the other hand, the Python implementation, +written in C, can completely hide implementation details and control +access to an object if necessary; this can be used by extensions to +Python written in C.) + + +Clients should use data attributes with care --- clients may mess up +invariants maintained by the methods by stamping on their data +attributes. Note that clients may add data attributes of their own to +an instance object without affecting the validity of the methods, as +long as name conflicts are avoided --- again, a naming convention can +save a lot of headaches here. + + +There is no shorthand for referencing data attributes (or other +methods!) from within methods. I find that this actually increases +the readability of methods: there is no chance of confusing local +variables and instance variables when glancing through a method. + + +Often, the first argument of a method is called +\code{self}. This is nothing more than a convention: the name +\code{self} has absolutely no special meaning to Python. (Note, +however, that by not following the convention your code may be less +readable to other Python programmers, and it is also conceivable that +a \emph{class browser} program might be written that relies upon such a +convention.) + + +Any function object that is a class attribute defines a method for +instances of that class. It is not necessary that the function +definition is textually enclosed in the class definition: assigning a +function object to a local variable in the class is also ok. For +example: + +\begin{verbatim} +# Function defined outside the class +def f1(self, x, y): + return min(x, x+y) + +class C: + f = f1 + def g(self): + return 'hello world' + h = g +\end{verbatim} + +Now \code{f}, \code{g} and \code{h} are all attributes of class +\class{C} that refer to function objects, and consequently they are all +methods of instances of \class{C} --- \code{h} being exactly equivalent +to \code{g}. Note that this practice usually only serves to confuse +the reader of a program. + + +Methods may call other methods by using method attributes of the +\code{self} argument: + +\begin{verbatim} +class Bag: + def __init__(self): + self.data = [] + def add(self, x): + self.data.append(x) + def addtwice(self, x): + self.add(x) + self.add(x) +\end{verbatim} + +Methods may reference global names in the same way as ordinary +functions. The global scope associated with a method is the module +containing the class definition. (The class itself is never used as a +global scope!) While one rarely encounters a good reason for using +global data in a method, there are many legitimate uses of the global +scope: for one thing, functions and modules imported into the global +scope can be used by methods, as well as functions and classes defined +in it. Usually, the class containing the method is itself defined in +this global scope, and in the next section we'll find some good +reasons why a method would want to reference its own class! + + +\section{Inheritance \label{inheritance}} + +Of course, a language feature would not be worthy of the name ``class'' +without supporting inheritance. The syntax for a derived class +definition looks like this: + +\begin{verbatim} +class DerivedClassName(BaseClassName): + <statement-1> + . + . + . + <statement-N> +\end{verbatim} + +The name \class{BaseClassName} must be defined in a scope containing +the derived class definition. In place of a base class name, other +arbitrary expressions are also allowed. This can be useful, for +example, when the base class is defined in another module: + +\begin{verbatim} +class DerivedClassName(modname.BaseClassName): +\end{verbatim} + +Execution of a derived class definition proceeds the same as for a +base class. When the class object is constructed, the base class is +remembered. This is used for resolving attribute references: if a +requested attribute is not found in the class, the search proceeds to look in the +base class. This rule is applied recursively if the base class itself +is derived from some other class. + +There's nothing special about instantiation of derived classes: +\code{DerivedClassName()} creates a new instance of the class. Method +references are resolved as follows: the corresponding class attribute +is searched, descending down the chain of base classes if necessary, +and the method reference is valid if this yields a function object. + +Derived classes may override methods of their base classes. Because +methods have no special privileges when calling other methods of the +same object, a method of a base class that calls another method +defined in the same base class may end up calling a method of +a derived class that overrides it. (For \Cpp{} programmers: all methods +in Python are effectively \keyword{virtual}.) + +An overriding method in a derived class may in fact want to extend +rather than simply replace the base class method of the same name. +There is a simple way to call the base class method directly: just +call \samp{BaseClassName.methodname(self, arguments)}. This is +occasionally useful to clients as well. (Note that this only works if +the base class is defined or imported directly in the global scope.) + + +\subsection{Multiple Inheritance \label{multiple}} + +Python supports a limited form of multiple inheritance as well. A +class definition with multiple base classes looks like this: + +\begin{verbatim} +class DerivedClassName(Base1, Base2, Base3): + <statement-1> + . + . + . + <statement-N> +\end{verbatim} + +For old-style classes, the only rule is depth-first, +left-to-right. Thus, if an attribute is not found in +\class{DerivedClassName}, it is searched in \class{Base1}, then +(recursively) in the base classes of \class{Base1}, and only if it is +not found there, it is searched in \class{Base2}, and so on. + +(To some people breadth first --- searching \class{Base2} and +\class{Base3} before the base classes of \class{Base1} --- looks more +natural. However, this would require you to know whether a particular +attribute of \class{Base1} is actually defined in \class{Base1} or in +one of its base classes before you can figure out the consequences of +a name conflict with an attribute of \class{Base2}. The depth-first +rule makes no differences between direct and inherited attributes of +\class{Base1}.) + +For new-style classes, the method resolution order changes dynamically +to support cooperative calls to \function{super()}. This approach +is known in some other multiple-inheritance languages as call-next-method +and is more powerful than the super call found in single-inheritance languages. + +With new-style classes, dynamic ordering is necessary because all +cases of multiple inheritance exhibit one or more diamond relationships +(where one at least one of the parent classes can be accessed through +multiple paths from the bottommost class). For example, all new-style +classes inherit from \class{object}, so any case of multiple inheritance +provides more than one path to reach \class{object}. To keep the +base classes from being accessed more than once, the dynamic algorithm +linearizes the search order in a way that preserves the left-to-right +ordering specified in each class, that calls each parent only once, and +that is monotonic (meaning that a class can be subclassed without affecting +the precedence order of its parents). Taken together, these properties +make it possible to design reliable and extensible classes with +multiple inheritance. For more detail, see +\url{http://www.python.org/download/releases/2.3/mro/}. + + +\section{Private Variables \label{private}} + +There is limited support for class-private +identifiers. Any identifier of the form \code{__spam} (at least two +leading underscores, at most one trailing underscore) is textually +replaced with \code{_classname__spam}, where \code{classname} is the +current class name with leading underscore(s) stripped. This mangling +is done without regard to the syntactic position of the identifier, so +it can be used to define class-private instance and class variables, +methods, variables stored in globals, and even variables stored in instances. +private to this class on instances of \emph{other} classes. Truncation +may occur when the mangled name would be longer than 255 characters. +Outside classes, or when the class name consists of only underscores, +no mangling occurs. + +Name mangling is intended to give classes an easy way to define +``private'' instance variables and methods, without having to worry +about instance variables defined by derived classes, or mucking with +instance variables by code outside the class. Note that the mangling +rules are designed mostly to avoid accidents; it still is possible for +a determined soul to access or modify a variable that is considered +private. This can even be useful in special circumstances, such as in +the debugger, and that's one reason why this loophole is not closed. +(Buglet: derivation of a class with the same name as the base class +makes use of private variables of the base class possible.) + +Notice that code passed to \code{exec}, \code{eval()} or +\code{execfile()} does not consider the classname of the invoking +class to be the current class; this is similar to the effect of the +\code{global} statement, the effect of which is likewise restricted to +code that is byte-compiled together. The same restriction applies to +\code{getattr()}, \code{setattr()} and \code{delattr()}, as well as +when referencing \code{__dict__} directly. + + +\section{Odds and Ends \label{odds}} + +Sometimes it is useful to have a data type similar to the Pascal +``record'' or C ``struct'', bundling together a few named data +items. An empty class definition will do nicely: + +\begin{verbatim} +class Employee: + pass + +john = Employee() # Create an empty employee record + +# Fill the fields of the record +john.name = 'John Doe' +john.dept = 'computer lab' +john.salary = 1000 +\end{verbatim} + +A piece of Python code that expects a particular abstract data type +can often be passed a class that emulates the methods of that data +type instead. For instance, if you have a function that formats some +data from a file object, you can define a class with methods +\method{read()} and \method{readline()} that get the data from a string +buffer instead, and pass it as an argument.% (Unfortunately, this +%technique has its limitations: a class can't define operations that +%are accessed by special syntax such as sequence subscripting or +%arithmetic operators, and assigning such a ``pseudo-file'' to +%\code{sys.stdin} will not cause the interpreter to read further input +%from it.) + + +Instance method objects have attributes, too: \code{m.im_self} is the +instance object with the method \method{m}, and \code{m.im_func} is the +function object corresponding to the method. + + +\section{Exceptions Are Classes Too\label{exceptionClasses}} + +User-defined exceptions are identified by classes as well. Using this +mechanism it is possible to create extensible hierarchies of exceptions. + +There are two new valid (semantic) forms for the raise statement: + +\begin{verbatim} +raise Class, instance + +raise instance +\end{verbatim} + +In the first form, \code{instance} must be an instance of +\class{Class} or of a class derived from it. The second form is a +shorthand for: + +\begin{verbatim} +raise instance.__class__, instance +\end{verbatim} + +A class in an except clause is compatible with an exception if it is the same +class or a base class thereof (but not the other way around --- an +except clause listing a derived class is not compatible with a base +class). For example, the following code will print B, C, D in that +order: + +\begin{verbatim} +class B: + pass +class C(B): + pass +class D(C): + pass + +for c in [B, C, D]: + try: + raise c() + except D: + print "D" + except C: + print "C" + except B: + print "B" +\end{verbatim} + +Note that if the except clauses were reversed (with +\samp{except B} first), it would have printed B, B, B --- the first +matching except clause is triggered. + +When an error message is printed for an unhandled exception, the +exception's class name is printed, then a colon and a space, and +finally the instance converted to a string using the built-in function +\function{str()}. + + +\section{Iterators\label{iterators}} + +By now you have probably noticed that most container objects can be looped +over using a \keyword{for} statement: + +\begin{verbatim} +for element in [1, 2, 3]: + print element +for element in (1, 2, 3): + print element +for key in {'one':1, 'two':2}: + print key +for char in "123": + print char +for line in open("myfile.txt"): + print line +\end{verbatim} + +This style of access is clear, concise, and convenient. The use of iterators +pervades and unifies Python. Behind the scenes, the \keyword{for} +statement calls \function{iter()} on the container object. The +function returns an iterator object that defines the method +\method{next()} which accesses elements in the container one at a +time. When there are no more elements, \method{next()} raises a +\exception{StopIteration} exception which tells the \keyword{for} loop +to terminate. This example shows how it all works: + +\begin{verbatim} +>>> s = 'abc' +>>> it = iter(s) +>>> it +<iterator object at 0x00A1DB50> +>>> it.next() +'a' +>>> it.next() +'b' +>>> it.next() +'c' +>>> it.next() + +Traceback (most recent call last): + File "<stdin>", line 1, in ? + it.next() +StopIteration +\end{verbatim} + +Having seen the mechanics behind the iterator protocol, it is easy to add +iterator behavior to your classes. Define a \method{__iter__()} method +which returns an object with a \method{next()} method. If the class defines +\method{next()}, then \method{__iter__()} can just return \code{self}: + +\begin{verbatim} +class Reverse: + "Iterator for looping over a sequence backwards" + def __init__(self, data): + self.data = data + self.index = len(data) + def __iter__(self): + return self + def next(self): + if self.index == 0: + raise StopIteration + self.index = self.index - 1 + return self.data[self.index] + +>>> for char in Reverse('spam'): +... print char +... +m +a +p +s +\end{verbatim} + + +\section{Generators\label{generators}} + +Generators are a simple and powerful tool for creating iterators. They are +written like regular functions but use the \keyword{yield} statement whenever +they want to return data. Each time \method{next()} is called, the +generator resumes where it left-off (it remembers all the data values and +which statement was last executed). An example shows that generators can +be trivially easy to create: + +\begin{verbatim} +def reverse(data): + for index in range(len(data)-1, -1, -1): + yield data[index] + +>>> for char in reverse('golf'): +... print char +... +f +l +o +g +\end{verbatim} + +Anything that can be done with generators can also be done with class based +iterators as described in the previous section. What makes generators so +compact is that the \method{__iter__()} and \method{next()} methods are +created automatically. + +Another key feature is that the local variables and execution state +are automatically saved between calls. This made the function easier to write +and much more clear than an approach using instance variables like +\code{self.index} and \code{self.data}. + +In addition to automatic method creation and saving program state, when +generators terminate, they automatically raise \exception{StopIteration}. +In combination, these features make it easy to create iterators with no +more effort than writing a regular function. + +\section{Generator Expressions\label{genexps}} + +Some simple generators can be coded succinctly as expressions using a syntax +similar to list comprehensions but with parentheses instead of brackets. These +expressions are designed for situations where the generator is used right +away by an enclosing function. Generator expressions are more compact but +less versatile than full generator definitions and tend to be more memory +friendly than equivalent list comprehensions. + +Examples: + +\begin{verbatim} +>>> sum(i*i for i in range(10)) # sum of squares +285 + +>>> xvec = [10, 20, 30] +>>> yvec = [7, 5, 3] +>>> sum(x*y for x,y in zip(xvec, yvec)) # dot product +260 + +>>> from math import pi, sin +>>> sine_table = dict((x, sin(x*pi/180)) for x in range(0, 91)) + +>>> unique_words = set(word for line in page for word in line.split()) + +>>> valedictorian = max((student.gpa, student.name) for student in graduates) + +>>> data = 'golf' +>>> list(data[i] for i in range(len(data)-1,-1,-1)) +['f', 'l', 'o', 'g'] + +\end{verbatim} + + + +\chapter{Brief Tour of the Standard Library \label{briefTour}} + + +\section{Operating System Interface\label{os-interface}} + +The \ulink{\module{os}}{../lib/module-os.html} +module provides dozens of functions for interacting with the +operating system: + +\begin{verbatim} +>>> import os +>>> os.system('time 0:02') +0 +>>> os.getcwd() # Return the current working directory +'C:\\Python24' +>>> os.chdir('/server/accesslogs') +\end{verbatim} + +Be sure to use the \samp{import os} style instead of +\samp{from os import *}. This will keep \function{os.open()} from +shadowing the builtin \function{open()} function which operates much +differently. + +\bifuncindex{help} +The builtin \function{dir()} and \function{help()} functions are useful +as interactive aids for working with large modules like \module{os}: + +\begin{verbatim} +>>> import os +>>> dir(os) +<returns a list of all module functions> +>>> help(os) +<returns an extensive manual page created from the module's docstrings> +\end{verbatim} + +For daily file and directory management tasks, the +\ulink{\module{shutil}}{../lib/module-shutil.html} +module provides a higher level interface that is easier to use: + +\begin{verbatim} +>>> import shutil +>>> shutil.copyfile('data.db', 'archive.db') +>>> shutil.move('/build/executables', 'installdir') +\end{verbatim} + + +\section{File Wildcards\label{file-wildcards}} + +The \ulink{\module{glob}}{../lib/module-glob.html} +module provides a function for making file lists from directory +wildcard searches: + +\begin{verbatim} +>>> import glob +>>> glob.glob('*.py') +['primes.py', 'random.py', 'quote.py'] +\end{verbatim} + + +\section{Command Line Arguments\label{command-line-arguments}} + +Common utility scripts often need to process command line arguments. +These arguments are stored in the +\ulink{\module{sys}}{../lib/module-sys.html}\ module's \var{argv} +attribute as a list. For instance the following output results from +running \samp{python demo.py one two three} at the command line: + +\begin{verbatim} +>>> import sys +>>> print sys.argv +['demo.py', 'one', 'two', 'three'] +\end{verbatim} + +The \ulink{\module{getopt}}{../lib/module-getopt.html} +module processes \var{sys.argv} using the conventions of the \UNIX{} +\function{getopt()} function. More powerful and flexible command line +processing is provided by the +\ulink{\module{optparse}}{../lib/module-optparse.html} module. + + +\section{Error Output Redirection and Program Termination\label{stderr}} + +The \ulink{\module{sys}}{../lib/module-sys.html} +module also has attributes for \var{stdin}, \var{stdout}, and +\var{stderr}. The latter is useful for emitting warnings and error +messages to make them visible even when \var{stdout} has been redirected: + +\begin{verbatim} +>>> sys.stderr.write('Warning, log file not found starting a new one\n') +Warning, log file not found starting a new one +\end{verbatim} + +The most direct way to terminate a script is to use \samp{sys.exit()}. + + +\section{String Pattern Matching\label{string-pattern-matching}} + +The \ulink{\module{re}}{../lib/module-re.html} +module provides regular expression tools for advanced string processing. +For complex matching and manipulation, regular expressions offer succinct, +optimized solutions: + +\begin{verbatim} +>>> import re +>>> re.findall(r'\bf[a-z]*', 'which foot or hand fell fastest') +['foot', 'fell', 'fastest'] +>>> re.sub(r'(\b[a-z]+) \1', r'\1', 'cat in the the hat') +'cat in the hat' +\end{verbatim} + +When only simple capabilities are needed, string methods are preferred +because they are easier to read and debug: + +\begin{verbatim} +>>> 'tea for too'.replace('too', 'two') +'tea for two' +\end{verbatim} + +\section{Mathematics\label{mathematics}} + +The \ulink{\module{math}}{../lib/module-math.html} module gives +access to the underlying C library functions for floating point math: + +\begin{verbatim} +>>> import math +>>> math.cos(math.pi / 4.0) +0.70710678118654757 +>>> math.log(1024, 2) +10.0 +\end{verbatim} + +The \ulink{\module{random}}{../lib/module-random.html} +module provides tools for making random selections: + +\begin{verbatim} +>>> import random +>>> random.choice(['apple', 'pear', 'banana']) +'apple' +>>> random.sample(xrange(100), 10) # sampling without replacement +[30, 83, 16, 4, 8, 81, 41, 50, 18, 33] +>>> random.random() # random float +0.17970987693706186 +>>> random.randrange(6) # random integer chosen from range(6) +4 +\end{verbatim} + + +\section{Internet Access\label{internet-access}} + +There are a number of modules for accessing the internet and processing +internet protocols. Two of the simplest are +\ulink{\module{urllib2}}{../lib/module-urllib2.html} +for retrieving data from urls and +\ulink{\module{smtplib}}{../lib/module-smtplib.html} +for sending mail: + +\begin{verbatim} +>>> import urllib2 +>>> for line in urllib2.urlopen('http://tycho.usno.navy.mil/cgi-bin/timer.pl'): +... if 'EST' in line or 'EDT' in line: # look for Eastern Time +... print line + +<BR>Nov. 25, 09:43:32 PM EST + +>>> import smtplib +>>> server = smtplib.SMTP('localhost') +>>> server.sendmail('soothsayer@example.org', 'jcaesar@example.org', +"""To: jcaesar@example.org +From: soothsayer@example.org + +Beware the Ides of March. +""") +>>> server.quit() +\end{verbatim} + + +\section{Dates and Times\label{dates-and-times}} + +The \ulink{\module{datetime}}{../lib/module-datetime.html} module +supplies classes for manipulating dates and times in both simple +and complex ways. While date and time arithmetic is supported, the +focus of the implementation is on efficient member extraction for +output formatting and manipulation. The module also supports objects +that are timezone aware. + +\begin{verbatim} +# dates are easily constructed and formatted +>>> from datetime import date +>>> now = date.today() +>>> now +datetime.date(2003, 12, 2) +>>> now.strftime("%m-%d-%y. %d %b %Y is a %A on the %d day of %B.") +'12-02-03. 02 Dec 2003 is a Tuesday on the 02 day of December.' + +# dates support calendar arithmetic +>>> birthday = date(1964, 7, 31) +>>> age = now - birthday +>>> age.days +14368 +\end{verbatim} + + +\section{Data Compression\label{data-compression}} + +Common data archiving and compression formats are directly supported +by modules including: +\ulink{\module{zlib}}{../lib/module-zlib.html}, +\ulink{\module{gzip}}{../lib/module-gzip.html}, +\ulink{\module{bz2}}{../lib/module-bz2.html}, +\ulink{\module{zipfile}}{../lib/module-zipfile.html}, and +\ulink{\module{tarfile}}{../lib/module-tarfile.html}. + +\begin{verbatim} +>>> import zlib +>>> s = 'witch which has which witches wrist watch' +>>> len(s) +41 +>>> t = zlib.compress(s) +>>> len(t) +37 +>>> zlib.decompress(t) +'witch which has which witches wrist watch' +>>> zlib.crc32(s) +226805979 +\end{verbatim} + + +\section{Performance Measurement\label{performance-measurement}} + +Some Python users develop a deep interest in knowing the relative +performance of different approaches to the same problem. +Python provides a measurement tool that answers those questions +immediately. + +For example, it may be tempting to use the tuple packing and unpacking +feature instead of the traditional approach to swapping arguments. +The \ulink{\module{timeit}}{../lib/module-timeit.html} module +quickly demonstrates a modest performance advantage: + +\begin{verbatim} +>>> from timeit import Timer +>>> Timer('t=a; a=b; b=t', 'a=1; b=2').timeit() +0.57535828626024577 +>>> Timer('a,b = b,a', 'a=1; b=2').timeit() +0.54962537085770791 +\end{verbatim} + +In contrast to \module{timeit}'s fine level of granularity, the +\ulink{\module{profile}}{../lib/module-profile.html} and \module{pstats} +modules provide tools for identifying time critical sections in larger blocks +of code. + + +\section{Quality Control\label{quality-control}} + +One approach for developing high quality software is to write tests for +each function as it is developed and to run those tests frequently during +the development process. + +The \ulink{\module{doctest}}{../lib/module-doctest.html} module provides +a tool for scanning a module and validating tests embedded in a program's +docstrings. Test construction is as simple as cutting-and-pasting a +typical call along with its results into the docstring. This improves +the documentation by providing the user with an example and it allows the +doctest module to make sure the code remains true to the documentation: + +\begin{verbatim} +def average(values): + """Computes the arithmetic mean of a list of numbers. + + >>> print average([20, 30, 70]) + 40.0 + """ + return sum(values, 0.0) / len(values) + +import doctest +doctest.testmod() # automatically validate the embedded tests +\end{verbatim} + +The \ulink{\module{unittest}}{../lib/module-unittest.html} module is not +as effortless as the \module{doctest} module, but it allows a more +comprehensive set of tests to be maintained in a separate file: + +\begin{verbatim} +import unittest + +class TestStatisticalFunctions(unittest.TestCase): + + def test_average(self): + self.assertEqual(average([20, 30, 70]), 40.0) + self.assertEqual(round(average([1, 5, 7]), 1), 4.3) + self.assertRaises(ZeroDivisionError, average, []) + self.assertRaises(TypeError, average, 20, 30, 70) + +unittest.main() # Calling from the command line invokes all tests +\end{verbatim} + +\section{Batteries Included\label{batteries-included}} + +Python has a ``batteries included'' philosophy. This is best seen +through the sophisticated and robust capabilities of its larger +packages. For example: + +\begin{itemize} +\item The \ulink{\module{xmlrpclib}}{../lib/module-xmlrpclib.html} and + \ulink{\module{SimpleXMLRPCServer}}{../lib/module-SimpleXMLRPCServer.html} + modules make implementing remote procedure calls into an almost trivial task. + Despite the modules names, no direct knowledge or handling of XML is needed. +\item The \ulink{\module{email}}{../lib/module-email.html} package is a library + for managing email messages, including MIME and other RFC 2822-based message + documents. Unlike \module{smtplib} and \module{poplib} which actually send + and receive messages, the email package has a complete toolset for building + or decoding complex message structures (including attachments) and for + implementing internet encoding and header protocols. +\item The \ulink{\module{xml.dom}}{../lib/module-xml.dom.html} and + \ulink{\module{xml.sax}}{../lib/module-xml.sax.html} packages provide robust + support for parsing this popular data interchange format. Likewise, the + \ulink{\module{csv}}{../lib/module-csv.html} module supports direct reads and + writes in a common database format. Together, these modules and packages + greatly simplify data interchange between python applications and other + tools. +\item Internationalization is supported by a number of modules including + \ulink{\module{gettext}}{../lib/module-gettext.html}, + \ulink{\module{locale}}{../lib/module-locale.html}, and the + \ulink{\module{codecs}}{../lib/module-codecs.html} package. +\end{itemize} + +\chapter{Brief Tour of the Standard Library -- Part II\label{briefTourTwo}} + +This second tour covers more advanced modules that support professional +programming needs. These modules rarely occur in small scripts. + + +\section{Output Formatting\label{output-formatting}} + +The \ulink{\module{repr}}{../lib/module-repr.html} module provides a +version of \function{repr()} customized for abbreviated displays of large +or deeply nested containers: + +\begin{verbatim} + >>> import repr + >>> repr.repr(set('supercalifragilisticexpialidocious')) + "set(['a', 'c', 'd', 'e', 'f', 'g', ...])" +\end{verbatim} + +The \ulink{\module{pprint}}{../lib/module-pprint.html} module offers +more sophisticated control over printing both built-in and user defined +objects in a way that is readable by the interpreter. When the result +is longer than one line, the ``pretty printer'' adds line breaks and +indentation to more clearly reveal data structure: + +\begin{verbatim} + >>> import pprint + >>> t = [[[['black', 'cyan'], 'white', ['green', 'red']], [['magenta', + ... 'yellow'], 'blue']]] + ... + >>> pprint.pprint(t, width=30) + [[[['black', 'cyan'], + 'white', + ['green', 'red']], + [['magenta', 'yellow'], + 'blue']]] +\end{verbatim} + +The \ulink{\module{textwrap}}{../lib/module-textwrap.html} module +formats paragraphs of text to fit a given screen width: + +\begin{verbatim} + >>> import textwrap + >>> doc = """The wrap() method is just like fill() except that it returns + ... a list of strings instead of one big string with newlines to separate + ... the wrapped lines.""" + ... + >>> print textwrap.fill(doc, width=40) + The wrap() method is just like fill() + except that it returns a list of strings + instead of one big string with newlines + to separate the wrapped lines. +\end{verbatim} + +The \ulink{\module{locale}}{../lib/module-locale.html} module accesses +a database of culture specific data formats. The grouping attribute +of locale's format function provides a direct way of formatting numbers +with group separators: + +\begin{verbatim} + >>> import locale + >>> locale.setlocale(locale.LC_ALL, 'English_United States.1252') + 'English_United States.1252' + >>> conv = locale.localeconv() # get a mapping of conventions + >>> x = 1234567.8 + >>> locale.format("%d", x, grouping=True) + '1,234,567' + >>> locale.format("%s%.*f", (conv['currency_symbol'], + ... conv['frac_digits'], x), grouping=True) + '$1,234,567.80' +\end{verbatim} + + +\section{Templating\label{templating}} + +The \ulink{\module{string}}{../lib/module-string.html} module includes a +versatile \class{Template} class with a simplified syntax suitable for +editing by end-users. This allows users to customize their applications +without having to alter the application. + +The format uses placeholder names formed by \samp{\$} with valid Python +identifiers (alphanumeric characters and underscores). Surrounding the +placeholder with braces allows it to be followed by more alphanumeric letters +with no intervening spaces. Writing \samp{\$\$} creates a single escaped +\samp{\$}: + +\begin{verbatim} +>>> from string import Template +>>> t = Template('${village}folk send $$10 to $cause.') +>>> t.substitute(village='Nottingham', cause='the ditch fund') +'Nottinghamfolk send $10 to the ditch fund.' +\end{verbatim} + +The \method{substitute} method raises a \exception{KeyError} when a +placeholder is not supplied in a dictionary or a keyword argument. For +mail-merge style applications, user supplied data may be incomplete and the +\method{safe_substitute} method may be more appropriate --- it will leave +placeholders unchanged if data is missing: + +\begin{verbatim} +>>> t = Template('Return the $item to $owner.') +>>> d = dict(item='unladen swallow') +>>> t.substitute(d) +Traceback (most recent call last): + . . . +KeyError: 'owner' +>>> t.safe_substitute(d) +'Return the unladen swallow to $owner.' +\end{verbatim} + +Template subclasses can specify a custom delimiter. For example, a batch +renaming utility for a photo browser may elect to use percent signs for +placeholders such as the current date, image sequence number, or file format: + +\begin{verbatim} +>>> import time, os.path +>>> photofiles = ['img_1074.jpg', 'img_1076.jpg', 'img_1077.jpg'] +>>> class BatchRename(Template): +... delimiter = '%' +>>> fmt = raw_input('Enter rename style (%d-date %n-seqnum %f-format): ') +Enter rename style (%d-date %n-seqnum %f-format): Ashley_%n%f + +>>> t = BatchRename(fmt) +>>> date = time.strftime('%d%b%y') +>>> for i, filename in enumerate(photofiles): +... base, ext = os.path.splitext(filename) +... newname = t.substitute(d=date, n=i, f=ext) +... print '%s --> %s' % (filename, newname) + +img_1074.jpg --> Ashley_0.jpg +img_1076.jpg --> Ashley_1.jpg +img_1077.jpg --> Ashley_2.jpg +\end{verbatim} + +Another application for templating is separating program logic from the +details of multiple output formats. This makes it possible to substitute +custom templates for XML files, plain text reports, and HTML web reports. + + +\section{Working with Binary Data Record Layouts\label{binary-formats}} + +The \ulink{\module{struct}}{../lib/module-struct.html} module provides +\function{pack()} and \function{unpack()} functions for working with +variable length binary record formats. The following example shows how +to loop through header information in a ZIP file (with pack codes +\code{"H"} and \code{"L"} representing two and four byte unsigned +numbers respectively): + +\begin{verbatim} + import struct + + data = open('myfile.zip', 'rb').read() + start = 0 + for i in range(3): # show the first 3 file headers + start += 14 + fields = struct.unpack('LLLHH', data[start:start+16]) + crc32, comp_size, uncomp_size, filenamesize, extra_size = fields + + start += 16 + filename = data[start:start+filenamesize] + start += filenamesize + extra = data[start:start+extra_size] + print filename, hex(crc32), comp_size, uncomp_size + + start += extra_size + comp_size # skip to the next header +\end{verbatim} + + +\section{Multi-threading\label{multi-threading}} + +Threading is a technique for decoupling tasks which are not sequentially +dependent. Threads can be used to improve the responsiveness of +applications that accept user input while other tasks run in the +background. A related use case is running I/O in parallel with +computations in another thread. + +The following code shows how the high level +\ulink{\module{threading}}{../lib/module-threading.html} module can run +tasks in background while the main program continues to run: + +\begin{verbatim} + import threading, zipfile + + class AsyncZip(threading.Thread): + def __init__(self, infile, outfile): + threading.Thread.__init__(self) + self.infile = infile + self.outfile = outfile + def run(self): + f = zipfile.ZipFile(self.outfile, 'w', zipfile.ZIP_DEFLATED) + f.write(self.infile) + f.close() + print 'Finished background zip of: ', self.infile + + background = AsyncZip('mydata.txt', 'myarchive.zip') + background.start() + print 'The main program continues to run in foreground.' + + background.join() # Wait for the background task to finish + print 'Main program waited until background was done.' +\end{verbatim} + +The principal challenge of multi-threaded applications is coordinating +threads that share data or other resources. To that end, the threading +module provides a number of synchronization primitives including locks, +events, condition variables, and semaphores. + +While those tools are powerful, minor design errors can result in +problems that are difficult to reproduce. So, the preferred approach +to task coordination is to concentrate all access to a resource +in a single thread and then use the +\ulink{\module{Queue}}{../lib/module-Queue.html} module to feed that +thread with requests from other threads. Applications using +\class{Queue} objects for inter-thread communication and coordination +are easier to design, more readable, and more reliable. + + +\section{Logging\label{logging}} + +The \ulink{\module{logging}}{../lib/module-logging.html} module offers +a full featured and flexible logging system. At its simplest, log +messages are sent to a file or to \code{sys.stderr}: + +\begin{verbatim} + import logging + logging.debug('Debugging information') + logging.info('Informational message') + logging.warning('Warning:config file %s not found', 'server.conf') + logging.error('Error occurred') + logging.critical('Critical error -- shutting down') +\end{verbatim} + +This produces the following output: + +\begin{verbatim} + WARNING:root:Warning:config file server.conf not found + ERROR:root:Error occurred + CRITICAL:root:Critical error -- shutting down +\end{verbatim} + +By default, informational and debugging messages are suppressed and the +output is sent to standard error. Other output options include routing +messages through email, datagrams, sockets, or to an HTTP Server. New +filters can select different routing based on message priority: +\constant{DEBUG}, \constant{INFO}, \constant{WARNING}, \constant{ERROR}, +and \constant{CRITICAL}. + +The logging system can be configured directly from Python or can be +loaded from a user editable configuration file for customized logging +without altering the application. + + +\section{Weak References\label{weak-references}} + +Python does automatic memory management (reference counting for most +objects and garbage collection to eliminate cycles). The memory is +freed shortly after the last reference to it has been eliminated. + +This approach works fine for most applications but occasionally there +is a need to track objects only as long as they are being used by +something else. Unfortunately, just tracking them creates a reference +that makes them permanent. The +\ulink{\module{weakref}}{../lib/module-weakref.html} module provides +tools for tracking objects without creating a reference. When the +object is no longer needed, it is automatically removed from a weakref +table and a callback is triggered for weakref objects. Typical +applications include caching objects that are expensive to create: + +\begin{verbatim} + >>> import weakref, gc + >>> class A: + ... def __init__(self, value): + ... self.value = value + ... def __repr__(self): + ... return str(self.value) + ... + >>> a = A(10) # create a reference + >>> d = weakref.WeakValueDictionary() + >>> d['primary'] = a # does not create a reference + >>> d['primary'] # fetch the object if it is still alive + 10 + >>> del a # remove the one reference + >>> gc.collect() # run garbage collection right away + 0 + >>> d['primary'] # entry was automatically removed + Traceback (most recent call last): + File "<pyshell#108>", line 1, in -toplevel- + d['primary'] # entry was automatically removed + File "C:/PY24/lib/weakref.py", line 46, in __getitem__ + o = self.data[key]() + KeyError: 'primary' +\end{verbatim} + +\section{Tools for Working with Lists\label{list-tools}} + +Many data structure needs can be met with the built-in list type. +However, sometimes there is a need for alternative implementations +with different performance trade-offs. + +The \ulink{\module{array}}{../lib/module-array.html} module provides an +\class{array()} object that is like a list that stores only homogenous +data and stores it more compactly. The following example shows an array +of numbers stored as two byte unsigned binary numbers (typecode +\code{"H"}) rather than the usual 16 bytes per entry for regular lists +of python int objects: + +\begin{verbatim} + >>> from array import array + >>> a = array('H', [4000, 10, 700, 22222]) + >>> sum(a) + 26932 + >>> a[1:3] + array('H', [10, 700]) +\end{verbatim} + +The \ulink{\module{collections}}{../lib/module-collections.html} module +provides a \class{deque()} object that is like a list with faster +appends and pops from the left side but slower lookups in the middle. +These objects are well suited for implementing queues and breadth first +tree searches: + +\begin{verbatim} + >>> from collections import deque + >>> d = deque(["task1", "task2", "task3"]) + >>> d.append("task4") + >>> print "Handling", d.popleft() + Handling task1 + + unsearched = deque([starting_node]) + def breadth_first_search(unsearched): + node = unsearched.popleft() + for m in gen_moves(node): + if is_goal(m): + return m + unsearched.append(m) +\end{verbatim} + +In addition to alternative list implementations, the library also offers +other tools such as the \ulink{\module{bisect}}{../lib/module-bisect.html} +module with functions for manipulating sorted lists: + +\begin{verbatim} + >>> import bisect + >>> scores = [(100, 'perl'), (200, 'tcl'), (400, 'lua'), (500, 'python')] + >>> bisect.insort(scores, (300, 'ruby')) + >>> scores + [(100, 'perl'), (200, 'tcl'), (300, 'ruby'), (400, 'lua'), (500, 'python')] +\end{verbatim} + +The \ulink{\module{heapq}}{../lib/module-heapq.html} module provides +functions for implementing heaps based on regular lists. The lowest +valued entry is always kept at position zero. This is useful for +applications which repeatedly access the smallest element but do not +want to run a full list sort: + +\begin{verbatim} + >>> from heapq import heapify, heappop, heappush + >>> data = [1, 3, 5, 7, 9, 2, 4, 6, 8, 0] + >>> heapify(data) # rearrange the list into heap order + >>> heappush(data, -5) # add a new entry + >>> [heappop(data) for i in range(3)] # fetch the three smallest entries + [-5, 0, 1] +\end{verbatim} + + +\section{Decimal Floating Point Arithmetic\label{decimal-fp}} + +The \ulink{\module{decimal}}{../lib/module-decimal.html} module offers a +\class{Decimal} datatype for decimal floating point arithmetic. Compared to +the built-in \class{float} implementation of binary floating point, the new +class is especially helpful for financial applications and other uses which +require exact decimal representation, control over precision, control over +rounding to meet legal or regulatory requirements, tracking of significant +decimal places, or for applications where the user expects the results to +match calculations done by hand. + +For example, calculating a 5\%{} tax on a 70 cent phone charge gives +different results in decimal floating point and binary floating point. +The difference becomes significant if the results are rounded to the +nearest cent: + +\begin{verbatim} +>>> from decimal import * +>>> Decimal('0.70') * Decimal('1.05') +Decimal("0.7350") +>>> .70 * 1.05 +0.73499999999999999 +\end{verbatim} + +The \class{Decimal} result keeps a trailing zero, automatically inferring four +place significance from multiplicands with two place significance. Decimal reproduces +mathematics as done by hand and avoids issues that can arise when binary +floating point cannot exactly represent decimal quantities. + +Exact representation enables the \class{Decimal} class to perform +modulo calculations and equality tests that are unsuitable for binary +floating point: + +\begin{verbatim} +>>> Decimal('1.00') % Decimal('.10') +Decimal("0.00") +>>> 1.00 % 0.10 +0.09999999999999995 + +>>> sum([Decimal('0.1')]*10) == Decimal('1.0') +True +>>> sum([0.1]*10) == 1.0 +False +\end{verbatim} + +The \module{decimal} module provides arithmetic with as much precision as +needed: + +\begin{verbatim} +>>> getcontext().prec = 36 +>>> Decimal(1) / Decimal(7) +Decimal("0.142857142857142857142857142857142857") +\end{verbatim} + + + +\chapter{What Now? \label{whatNow}} + +Reading this tutorial has probably reinforced your interest in using +Python --- you should be eager to apply Python to solving your +real-world problems. Where should you go to learn more? + +This tutorial is part of Python's documentation set. +Some other documents in the set are: + +\begin{itemize} + +\item \citetitle[../lib/lib.html]{Python Library Reference}: + +You should browse through this manual, which gives complete (though +terse) reference material about types, functions, and the modules in +the standard library. The standard Python distribution includes a +\emph{lot} of additional code. There are modules to read \UNIX{} +mailboxes, retrieve documents via HTTP, generate random numbers, parse +command-line options, write CGI programs, compress data, and many other tasks. +Skimming through the Library Reference will give you an idea of +what's available. + +\item \citetitle[../inst/inst.html]{Installing Python Modules} +explains how to install external modules written by other Python +users. + +\item \citetitle[../ref/ref.html]{Language Reference}: A detailed +explanation of Python's syntax and semantics. It's heavy reading, +but is useful as a complete guide to the language itself. + +\end{itemize} + +More Python resources: + +\begin{itemize} + +\item \url{http://www.python.org}: The major Python Web site. It contains +code, documentation, and pointers to Python-related pages around the +Web. This Web site is mirrored in various places around the +world, such as Europe, Japan, and Australia; a mirror may be faster +than the main site, depending on your geographical location. + +\item \url{http://docs.python.org}: Fast access to Python's +documentation. + +\item \url{http://cheeseshop.python.org}: +The Python Package Index, nicknamed the Cheese Shop, +is an index of user-created Python modules that are available for +download. Once you begin releasing code, you can register it +here so that others can find it. + +\item \url{http://aspn.activestate.com/ASPN/Python/Cookbook/}: The +Python Cookbook is a sizable collection of code examples, larger +modules, and useful scripts. Particularly notable contributions are +collected in a book also titled \citetitle{Python Cookbook} (O'Reilly +\& Associates, ISBN 0-596-00797-3.) + +\end{itemize} + + +For Python-related questions and problem reports, you can post to the +newsgroup \newsgroup{comp.lang.python}, or send them to the mailing +list at \email{python-list@python.org}. The newsgroup and mailing list +are gatewayed, so messages posted to one will automatically be +forwarded to the other. There are around 120 postings a day (with peaks +up to several hundred), +% Postings figure based on average of last six months activity as +% reported by www.egroups.com; Jan. 2000 - June 2000: 21272 msgs / 182 +% days = 116.9 msgs / day and steadily increasing. +asking (and answering) questions, suggesting new features, and +announcing new modules. Before posting, be sure to check the list of +\ulink{Frequently Asked Questions}{http://www.python.org/doc/faq/} (also called the FAQ), or look for it in the +\file{Misc/} directory of the Python source distribution. Mailing +list archives are available at \url{http://mail.python.org/pipermail/}. +The FAQ answers many of the questions that come up again and again, +and may already contain the solution for your problem. + + +\appendix + +\chapter{Interactive Input Editing and History Substitution\label{interacting}} + +Some versions of the Python interpreter support editing of the current +input line and history substitution, similar to facilities found in +the Korn shell and the GNU Bash shell. This is implemented using the +\emph{GNU Readline} library, which supports Emacs-style and vi-style +editing. This library has its own documentation which I won't +duplicate here; however, the basics are easily explained. The +interactive editing and history described here are optionally +available in the \UNIX{} and Cygwin versions of the interpreter. + +This chapter does \emph{not} document the editing facilities of Mark +Hammond's PythonWin package or the Tk-based environment, IDLE, +distributed with Python. The command line history recall which +operates within DOS boxes on NT and some other DOS and Windows flavors +is yet another beast. + +\section{Line Editing \label{lineEditing}} + +If supported, input line editing is active whenever the interpreter +prints a primary or secondary prompt. The current line can be edited +using the conventional Emacs control characters. The most important +of these are: \kbd{C-A} (Control-A) moves the cursor to the beginning +of the line, \kbd{C-E} to the end, \kbd{C-B} moves it one position to +the left, \kbd{C-F} to the right. Backspace erases the character to +the left of the cursor, \kbd{C-D} the character to its right. +\kbd{C-K} kills (erases) the rest of the line to the right of the +cursor, \kbd{C-Y} yanks back the last killed string. +\kbd{C-underscore} undoes the last change you made; it can be repeated +for cumulative effect. + +\section{History Substitution \label{history}} + +History substitution works as follows. All non-empty input lines +issued are saved in a history buffer, and when a new prompt is given +you are positioned on a new line at the bottom of this buffer. +\kbd{C-P} moves one line up (back) in the history buffer, +\kbd{C-N} moves one down. Any line in the history buffer can be +edited; an asterisk appears in front of the prompt to mark a line as +modified. Pressing the \kbd{Return} key passes the current line to +the interpreter. \kbd{C-R} starts an incremental reverse search; +\kbd{C-S} starts a forward search. + +\section{Key Bindings \label{keyBindings}} + +The key bindings and some other parameters of the Readline library can +be customized by placing commands in an initialization file called +\file{\~{}/.inputrc}. Key bindings have the form + +\begin{verbatim} +key-name: function-name +\end{verbatim} + +or + +\begin{verbatim} +"string": function-name +\end{verbatim} + +and options can be set with + +\begin{verbatim} +set option-name value +\end{verbatim} + +For example: + +\begin{verbatim} +# I prefer vi-style editing: +set editing-mode vi + +# Edit using a single line: +set horizontal-scroll-mode On + +# Rebind some keys: +Meta-h: backward-kill-word +"\C-u": universal-argument +"\C-x\C-r": re-read-init-file +\end{verbatim} + +Note that the default binding for \kbd{Tab} in Python is to insert a +\kbd{Tab} character instead of Readline's default filename completion +function. If you insist, you can override this by putting + +\begin{verbatim} +Tab: complete +\end{verbatim} + +in your \file{\~{}/.inputrc}. (Of course, this makes it harder to +type indented continuation lines if you're accustomed to using +\kbd{Tab} for that purpose.) + +Automatic completion of variable and module names is optionally +available. To enable it in the interpreter's interactive mode, add +the following to your startup file:\footnote{ + Python will execute the contents of a file identified by the + \envvar{PYTHONSTARTUP} environment variable when you start an + interactive interpreter.} +\refstmodindex{rlcompleter}\refbimodindex{readline} + +\begin{verbatim} +import rlcompleter, readline +readline.parse_and_bind('tab: complete') +\end{verbatim} + +This binds the \kbd{Tab} key to the completion function, so hitting +the \kbd{Tab} key twice suggests completions; it looks at Python +statement names, the current local variables, and the available module +names. For dotted expressions such as \code{string.a}, it will +evaluate the expression up to the final \character{.} and then +suggest completions from the attributes of the resulting object. Note +that this may execute application-defined code if an object with a +\method{__getattr__()} method is part of the expression. + +A more capable startup file might look like this example. Note that +this deletes the names it creates once they are no longer needed; this +is done since the startup file is executed in the same namespace as +the interactive commands, and removing the names avoids creating side +effects in the interactive environment. You may find it convenient +to keep some of the imported modules, such as +\ulink{\module{os}}{../lib/module-os.html}, which turn +out to be needed in most sessions with the interpreter. + +\begin{verbatim} +# Add auto-completion and a stored history file of commands to your Python +# interactive interpreter. Requires Python 2.0+, readline. Autocomplete is +# bound to the Esc key by default (you can change it - see readline docs). +# +# Store the file in ~/.pystartup, and set an environment variable to point +# to it: "export PYTHONSTARTUP=/max/home/itamar/.pystartup" in bash. +# +# Note that PYTHONSTARTUP does *not* expand "~", so you have to put in the +# full path to your home directory. + +import atexit +import os +import readline +import rlcompleter + +historyPath = os.path.expanduser("~/.pyhistory") + +def save_history(historyPath=historyPath): + import readline + readline.write_history_file(historyPath) + +if os.path.exists(historyPath): + readline.read_history_file(historyPath) + +atexit.register(save_history) +del os, atexit, readline, rlcompleter, save_history, historyPath +\end{verbatim} + + +\section{Commentary \label{commentary}} + +This facility is an enormous step forward compared to earlier versions +of the interpreter; however, some wishes are left: It would be nice if +the proper indentation were suggested on continuation lines (the +parser knows if an indent token is required next). The completion +mechanism might use the interpreter's symbol table. A command to +check (or even suggest) matching parentheses, quotes, etc., would also +be useful. + + +\chapter{Floating Point Arithmetic: Issues and Limitations\label{fp-issues}} +\sectionauthor{Tim Peters}{tim_one@users.sourceforge.net} + +Floating-point numbers are represented in computer hardware as +base 2 (binary) fractions. For example, the decimal fraction + +\begin{verbatim} +0.125 +\end{verbatim} + +has value 1/10 + 2/100 + 5/1000, and in the same way the binary fraction + +\begin{verbatim} +0.001 +\end{verbatim} + +has value 0/2 + 0/4 + 1/8. These two fractions have identical values, +the only real difference being that the first is written in base 10 +fractional notation, and the second in base 2. + +Unfortunately, most decimal fractions cannot be represented exactly as +binary fractions. A consequence is that, in general, the decimal +floating-point numbers you enter are only approximated by the binary +floating-point numbers actually stored in the machine. + +The problem is easier to understand at first in base 10. Consider the +fraction 1/3. You can approximate that as a base 10 fraction: + +\begin{verbatim} +0.3 +\end{verbatim} + +or, better, + +\begin{verbatim} +0.33 +\end{verbatim} + +or, better, + +\begin{verbatim} +0.333 +\end{verbatim} + +and so on. No matter how many digits you're willing to write down, the +result will never be exactly 1/3, but will be an increasingly better +approximation of 1/3. + +In the same way, no matter how many base 2 digits you're willing to +use, the decimal value 0.1 cannot be represented exactly as a base 2 +fraction. In base 2, 1/10 is the infinitely repeating fraction + +\begin{verbatim} +0.0001100110011001100110011001100110011001100110011... +\end{verbatim} + +Stop at any finite number of bits, and you get an approximation. This +is why you see things like: + +\begin{verbatim} +>>> 0.1 +0.10000000000000001 +\end{verbatim} + +On most machines today, that is what you'll see if you enter 0.1 at +a Python prompt. You may not, though, because the number of bits +used by the hardware to store floating-point values can vary across +machines, and Python only prints a decimal approximation to the true +decimal value of the binary approximation stored by the machine. On +most machines, if Python were to print the true decimal value of +the binary approximation stored for 0.1, it would have to display + +\begin{verbatim} +>>> 0.1 +0.1000000000000000055511151231257827021181583404541015625 +\end{verbatim} + +instead! The Python prompt uses the builtin +\function{repr()} function to obtain a string version of everything it +displays. For floats, \code{repr(\var{float})} rounds the true +decimal value to 17 significant digits, giving + +\begin{verbatim} +0.10000000000000001 +\end{verbatim} + +\code{repr(\var{float})} produces 17 significant digits because it +turns out that's enough (on most machines) so that +\code{eval(repr(\var{x})) == \var{x}} exactly for all finite floats +\var{x}, but rounding to 16 digits is not enough to make that true. + +Note that this is in the very nature of binary floating-point: this is +not a bug in Python, and it is not a bug in your code either. You'll +see the same kind of thing in all languages that support your +hardware's floating-point arithmetic (although some languages may +not \emph{display} the difference by default, or in all output modes). + +Python's builtin \function{str()} function produces only 12 +significant digits, and you may wish to use that instead. It's +unusual for \code{eval(str(\var{x}))} to reproduce \var{x}, but the +output may be more pleasant to look at: + +\begin{verbatim} +>>> print str(0.1) +0.1 +\end{verbatim} + +It's important to realize that this is, in a real sense, an illusion: +the value in the machine is not exactly 1/10, you're simply rounding +the \emph{display} of the true machine value. + +Other surprises follow from this one. For example, after seeing + +\begin{verbatim} +>>> 0.1 +0.10000000000000001 +\end{verbatim} + +you may be tempted to use the \function{round()} function to chop it +back to the single digit you expect. But that makes no difference: + +\begin{verbatim} +>>> round(0.1, 1) +0.10000000000000001 +\end{verbatim} + +The problem is that the binary floating-point value stored for "0.1" +was already the best possible binary approximation to 1/10, so trying +to round it again can't make it better: it was already as good as it +gets. + +Another consequence is that since 0.1 is not exactly 1/10, +summing ten values of 0.1 may not yield exactly 1.0, either: + +\begin{verbatim} +>>> sum = 0.0 +>>> for i in range(10): +... sum += 0.1 +... +>>> sum +0.99999999999999989 +\end{verbatim} + +Binary floating-point arithmetic holds many surprises like this. The +problem with "0.1" is explained in precise detail below, in the +"Representation Error" section. See +\citetitle[http://www.lahey.com/float.htm]{The Perils of Floating +Point} for a more complete account of other common surprises. + +As that says near the end, ``there are no easy answers.'' Still, +don't be unduly wary of floating-point! The errors in Python float +operations are inherited from the floating-point hardware, and on most +machines are on the order of no more than 1 part in 2**53 per +operation. That's more than adequate for most tasks, but you do need +to keep in mind that it's not decimal arithmetic, and that every float +operation can suffer a new rounding error. + +While pathological cases do exist, for most casual use of +floating-point arithmetic you'll see the result you expect in the end +if you simply round the display of your final results to the number of +decimal digits you expect. \function{str()} usually suffices, and for +finer control see the discussion of Python's \code{\%} format +operator: the \code{\%g}, \code{\%f} and \code{\%e} format codes +supply flexible and easy ways to round float results for display. + + +\section{Representation Error + \label{fp-error}} + +This section explains the ``0.1'' example in detail, and shows how +you can perform an exact analysis of cases like this yourself. Basic +familiarity with binary floating-point representation is assumed. + +\dfn{Representation error} refers to the fact that some (most, actually) +decimal fractions cannot be represented exactly as binary (base 2) +fractions. This is the chief reason why Python (or Perl, C, \Cpp, +Java, Fortran, and many others) often won't display the exact decimal +number you expect: + +\begin{verbatim} +>>> 0.1 +0.10000000000000001 +\end{verbatim} + +Why is that? 1/10 is not exactly representable as a binary fraction. +Almost all machines today (November 2000) use IEEE-754 floating point +arithmetic, and almost all platforms map Python floats to IEEE-754 +"double precision". 754 doubles contain 53 bits of precision, so on +input the computer strives to convert 0.1 to the closest fraction it can +of the form \var{J}/2**\var{N} where \var{J} is an integer containing +exactly 53 bits. Rewriting + +\begin{verbatim} + 1 / 10 ~= J / (2**N) +\end{verbatim} + +as + +\begin{verbatim} +J ~= 2**N / 10 +\end{verbatim} + +and recalling that \var{J} has exactly 53 bits (is \code{>= 2**52} but +\code{< 2**53}), the best value for \var{N} is 56: + +\begin{verbatim} +>>> 2**52 +4503599627370496L +>>> 2**53 +9007199254740992L +>>> 2**56/10 +7205759403792793L +\end{verbatim} + +That is, 56 is the only value for \var{N} that leaves \var{J} with +exactly 53 bits. The best possible value for \var{J} is then that +quotient rounded: + +\begin{verbatim} +>>> q, r = divmod(2**56, 10) +>>> r +6L +\end{verbatim} + +Since the remainder is more than half of 10, the best approximation is +obtained by rounding up: + +\begin{verbatim} +>>> q+1 +7205759403792794L +\end{verbatim} + +Therefore the best possible approximation to 1/10 in 754 double +precision is that over 2**56, or + +\begin{verbatim} +7205759403792794 / 72057594037927936 +\end{verbatim} + +Note that since we rounded up, this is actually a little bit larger than +1/10; if we had not rounded up, the quotient would have been a little +bit smaller than 1/10. But in no case can it be \emph{exactly} 1/10! + +So the computer never ``sees'' 1/10: what it sees is the exact +fraction given above, the best 754 double approximation it can get: + +\begin{verbatim} +>>> .1 * 2**56 +7205759403792794.0 +\end{verbatim} + +If we multiply that fraction by 10**30, we can see the (truncated) +value of its 30 most significant decimal digits: + +\begin{verbatim} +>>> 7205759403792794 * 10**30 / 2**56 +100000000000000005551115123125L +\end{verbatim} + +meaning that the exact number stored in the computer is approximately +equal to the decimal value 0.100000000000000005551115123125. Rounding +that to 17 significant digits gives the 0.10000000000000001 that Python +displays (well, will display on any 754-conforming platform that does +best-possible input and output conversions in its C library --- yours may +not!). + +\chapter{History and License} +\input{license} + +\input{glossary} + +\input{tut.ind} + +\end{document} |