diff options
author | cinap_lenrek <cinap_lenrek@localhost> | 2011-05-03 11:25:13 +0000 |
---|---|---|
committer | cinap_lenrek <cinap_lenrek@localhost> | 2011-05-03 11:25:13 +0000 |
commit | 458120dd40db6b4df55a4e96b650e16798ef06a0 (patch) | |
tree | 8f82685be24fef97e715c6f5ca4c68d34d5074ee /sys/src/cmd/python/Doc/lib/libdoctest.tex | |
parent | 3a742c699f6806c1145aea5149bf15de15a0afd7 (diff) |
add hg and python
Diffstat (limited to 'sys/src/cmd/python/Doc/lib/libdoctest.tex')
-rw-r--r-- | sys/src/cmd/python/Doc/lib/libdoctest.tex | 1949 |
1 files changed, 1949 insertions, 0 deletions
diff --git a/sys/src/cmd/python/Doc/lib/libdoctest.tex b/sys/src/cmd/python/Doc/lib/libdoctest.tex new file mode 100644 index 000000000..957ecf48c --- /dev/null +++ b/sys/src/cmd/python/Doc/lib/libdoctest.tex @@ -0,0 +1,1949 @@ +\section{\module{doctest} --- + Test interactive Python examples} + +\declaremodule{standard}{doctest} +\moduleauthor{Tim Peters}{tim@python.org} +\sectionauthor{Tim Peters}{tim@python.org} +\sectionauthor{Moshe Zadka}{moshez@debian.org} +\sectionauthor{Edward Loper}{edloper@users.sourceforge.net} + +\modulesynopsis{A framework for verifying interactive Python examples.} + +The \refmodule{doctest} module searches for pieces of text that look like +interactive Python sessions, and then executes those sessions to +verify that they work exactly as shown. There are several common ways to +use doctest: + +\begin{itemize} +\item To check that a module's docstrings are up-to-date by verifying + that all interactive examples still work as documented. +\item To perform regression testing by verifying that interactive + examples from a test file or a test object work as expected. +\item To write tutorial documentation for a package, liberally + illustrated with input-output examples. Depending on whether + the examples or the expository text are emphasized, this has + the flavor of "literate testing" or "executable documentation". +\end{itemize} + +Here's a complete but small example module: + +\begin{verbatim} +""" +This is the "example" module. + +The example module supplies one function, factorial(). For example, + +>>> factorial(5) +120 +""" + +def factorial(n): + """Return the factorial of n, an exact integer >= 0. + + If the result is small enough to fit in an int, return an int. + Else return a long. + + >>> [factorial(n) for n in range(6)] + [1, 1, 2, 6, 24, 120] + >>> [factorial(long(n)) for n in range(6)] + [1, 1, 2, 6, 24, 120] + >>> factorial(30) + 265252859812191058636308480000000L + >>> factorial(30L) + 265252859812191058636308480000000L + >>> factorial(-1) + Traceback (most recent call last): + ... + ValueError: n must be >= 0 + + Factorials of floats are OK, but the float must be an exact integer: + >>> factorial(30.1) + Traceback (most recent call last): + ... + ValueError: n must be exact integer + >>> factorial(30.0) + 265252859812191058636308480000000L + + It must also not be ridiculously large: + >>> factorial(1e100) + Traceback (most recent call last): + ... + OverflowError: n too large + """ + +\end{verbatim} +% allow LaTeX to break here. +\begin{verbatim} + + import math + if not n >= 0: + raise ValueError("n must be >= 0") + if math.floor(n) != n: + raise ValueError("n must be exact integer") + if n+1 == n: # catch a value like 1e300 + raise OverflowError("n too large") + result = 1 + factor = 2 + while factor <= n: + result *= factor + factor += 1 + return result + +def _test(): + import doctest + doctest.testmod() + +if __name__ == "__main__": + _test() +\end{verbatim} + +If you run \file{example.py} directly from the command line, +\refmodule{doctest} works its magic: + +\begin{verbatim} +$ python example.py +$ +\end{verbatim} + +There's no output! That's normal, and it means all the examples +worked. Pass \programopt{-v} to the script, and \refmodule{doctest} +prints a detailed log of what it's trying, and prints a summary at the +end: + +\begin{verbatim} +$ python example.py -v +Trying: + factorial(5) +Expecting: + 120 +ok +Trying: + [factorial(n) for n in range(6)] +Expecting: + [1, 1, 2, 6, 24, 120] +ok +Trying: + [factorial(long(n)) for n in range(6)] +Expecting: + [1, 1, 2, 6, 24, 120] +ok +\end{verbatim} + +And so on, eventually ending with: + +\begin{verbatim} +Trying: + factorial(1e100) +Expecting: + Traceback (most recent call last): + ... + OverflowError: n too large +ok +1 items had no tests: + __main__._test +2 items passed all tests: + 1 tests in __main__ + 8 tests in __main__.factorial +9 tests in 3 items. +9 passed and 0 failed. +Test passed. +$ +\end{verbatim} + +That's all you need to know to start making productive use of +\refmodule{doctest}! Jump in. The following sections provide full +details. Note that there are many examples of doctests in +the standard Python test suite and libraries. Especially useful examples +can be found in the standard test file \file{Lib/test/test_doctest.py}. + +\subsection{Simple Usage: Checking Examples in + Docstrings\label{doctest-simple-testmod}} + +The simplest way to start using doctest (but not necessarily the way +you'll continue to do it) is to end each module \module{M} with: + +\begin{verbatim} +def _test(): + import doctest + doctest.testmod() + +if __name__ == "__main__": + _test() +\end{verbatim} + +\refmodule{doctest} then examines docstrings in module \module{M}. + +Running the module as a script causes the examples in the docstrings +to get executed and verified: + +\begin{verbatim} +python M.py +\end{verbatim} + +This won't display anything unless an example fails, in which case the +failing example(s) and the cause(s) of the failure(s) are printed to stdout, +and the final line of output is +\samp{***Test Failed*** \var{N} failures.}, where \var{N} is the +number of examples that failed. + +Run it with the \programopt{-v} switch instead: + +\begin{verbatim} +python M.py -v +\end{verbatim} + +and a detailed report of all examples tried is printed to standard +output, along with assorted summaries at the end. + +You can force verbose mode by passing \code{verbose=True} to +\function{testmod()}, or +prohibit it by passing \code{verbose=False}. In either of those cases, +\code{sys.argv} is not examined by \function{testmod()} (so passing +\programopt{-v} or not has no effect). + +For more information on \function{testmod()}, see +section~\ref{doctest-basic-api}. + +\subsection{Simple Usage: Checking Examples in a Text + File\label{doctest-simple-testfile}} + +Another simple application of doctest is testing interactive examples +in a text file. This can be done with the \function{testfile()} +function: + +\begin{verbatim} +import doctest +doctest.testfile("example.txt") +\end{verbatim} + +That short script executes and verifies any interactive Python +examples contained in the file \file{example.txt}. The file content +is treated as if it were a single giant docstring; the file doesn't +need to contain a Python program! For example, perhaps \file{example.txt} +contains this: + +\begin{verbatim} +The ``example`` module +====================== + +Using ``factorial`` +------------------- + +This is an example text file in reStructuredText format. First import +``factorial`` from the ``example`` module: + + >>> from example import factorial + +Now use it: + + >>> factorial(6) + 120 +\end{verbatim} + +Running \code{doctest.testfile("example.txt")} then finds the error +in this documentation: + +\begin{verbatim} +File "./example.txt", line 14, in example.txt +Failed example: + factorial(6) +Expected: + 120 +Got: + 720 +\end{verbatim} + +As with \function{testmod()}, \function{testfile()} won't display anything +unless an example fails. If an example does fail, then the failing +example(s) and the cause(s) of the failure(s) are printed to stdout, using +the same format as \function{testmod()}. + +By default, \function{testfile()} looks for files in the calling +module's directory. See section~\ref{doctest-basic-api} for a +description of the optional arguments that can be used to tell it to +look for files in other locations. + +Like \function{testmod()}, \function{testfile()}'s verbosity can be +set with the \programopt{-v} command-line switch or with the optional +keyword argument \var{verbose}. + +For more information on \function{testfile()}, see +section~\ref{doctest-basic-api}. + +\subsection{How It Works\label{doctest-how-it-works}} + +This section examines in detail how doctest works: which docstrings it +looks at, how it finds interactive examples, what execution context it +uses, how it handles exceptions, and how option flags can be used to +control its behavior. This is the information that you need to know +to write doctest examples; for information about actually running +doctest on these examples, see the following sections. + +\subsubsection{Which Docstrings Are Examined?\label{doctest-which-docstrings}} + +The module docstring, and all function, class and method docstrings are +searched. Objects imported into the module are not searched. + +In addition, if \code{M.__test__} exists and "is true", it must be a +dict, and each entry maps a (string) name to a function object, class +object, or string. Function and class object docstrings found from +\code{M.__test__} are searched, and strings are treated as if they +were docstrings. In output, a key \code{K} in \code{M.__test__} appears +with name + +\begin{verbatim} +<name of M>.__test__.K +\end{verbatim} + +Any classes found are recursively searched similarly, to test docstrings in +their contained methods and nested classes. + +\versionchanged[A "private name" concept is deprecated and no longer + documented]{2.4} + +\subsubsection{How are Docstring Examples + Recognized?\label{doctest-finding-examples}} + +In most cases a copy-and-paste of an interactive console session works +fine, but doctest isn't trying to do an exact emulation of any specific +Python shell. All hard tab characters are expanded to spaces, using +8-column tab stops. If you don't believe tabs should mean that, too +bad: don't use hard tabs, or write your own \class{DocTestParser} +class. + +\versionchanged[Expanding tabs to spaces is new; previous versions + tried to preserve hard tabs, with confusing results]{2.4} + +\begin{verbatim} +>>> # comments are ignored +>>> x = 12 +>>> x +12 +>>> if x == 13: +... print "yes" +... else: +... print "no" +... print "NO" +... print "NO!!!" +... +no +NO +NO!!! +>>> +\end{verbatim} + +Any expected output must immediately follow the final +\code{'>>>~'} or \code{'...~'} line containing the code, and +the expected output (if any) extends to the next \code{'>>>~'} +or all-whitespace line. + +The fine print: + +\begin{itemize} + +\item Expected output cannot contain an all-whitespace line, since such a + line is taken to signal the end of expected output. If expected + output does contain a blank line, put \code{<BLANKLINE>} in your + doctest example each place a blank line is expected. + \versionchanged[\code{<BLANKLINE>} was added; there was no way to + use expected output containing empty lines in + previous versions]{2.4} + +\item Output to stdout is captured, but not output to stderr (exception + tracebacks are captured via a different means). + +\item If you continue a line via backslashing in an interactive session, + or for any other reason use a backslash, you should use a raw + docstring, which will preserve your backslashes exactly as you type + them: + +\begin{verbatim} +>>> def f(x): +... r'''Backslashes in a raw docstring: m\n''' +>>> print f.__doc__ +Backslashes in a raw docstring: m\n +\end{verbatim} + + Otherwise, the backslash will be interpreted as part of the string. + For example, the "{\textbackslash}" above would be interpreted as a + newline character. Alternatively, you can double each backslash in the + doctest version (and not use a raw string): + +\begin{verbatim} +>>> def f(x): +... '''Backslashes in a raw docstring: m\\n''' +>>> print f.__doc__ +Backslashes in a raw docstring: m\n +\end{verbatim} + +\item The starting column doesn't matter: + +\begin{verbatim} + >>> assert "Easy!" + >>> import math + >>> math.floor(1.9) + 1.0 +\end{verbatim} + +and as many leading whitespace characters are stripped from the +expected output as appeared in the initial \code{'>>>~'} line +that started the example. +\end{itemize} + +\subsubsection{What's the Execution Context?\label{doctest-execution-context}} + +By default, each time \refmodule{doctest} finds a docstring to test, it +uses a \emph{shallow copy} of \module{M}'s globals, so that running tests +doesn't change the module's real globals, and so that one test in +\module{M} can't leave behind crumbs that accidentally allow another test +to work. This means examples can freely use any names defined at top-level +in \module{M}, and names defined earlier in the docstring being run. +Examples cannot see names defined in other docstrings. + +You can force use of your own dict as the execution context by passing +\code{globs=your_dict} to \function{testmod()} or +\function{testfile()} instead. + +\subsubsection{What About Exceptions?\label{doctest-exceptions}} + +No problem, provided that the traceback is the only output produced by +the example: just paste in the traceback.\footnote{Examples containing + both expected output and an exception are not supported. Trying + to guess where one ends and the other begins is too error-prone, + and that also makes for a confusing test.} +Since tracebacks contain details that are likely to change rapidly (for +example, exact file paths and line numbers), this is one case where doctest +works hard to be flexible in what it accepts. + +Simple example: + +\begin{verbatim} +>>> [1, 2, 3].remove(42) +Traceback (most recent call last): + File "<stdin>", line 1, in ? +ValueError: list.remove(x): x not in list +\end{verbatim} + +That doctest succeeds if \exception{ValueError} is raised, with the +\samp{list.remove(x): x not in list} detail as shown. + +The expected output for an exception must start with a traceback +header, which may be either of the following two lines, indented the +same as the first line of the example: + +\begin{verbatim} +Traceback (most recent call last): +Traceback (innermost last): +\end{verbatim} + +The traceback header is followed by an optional traceback stack, whose +contents are ignored by doctest. The traceback stack is typically +omitted, or copied verbatim from an interactive session. + +The traceback stack is followed by the most interesting part: the +line(s) containing the exception type and detail. This is usually the +last line of a traceback, but can extend across multiple lines if the +exception has a multi-line detail: + +\begin{verbatim} +>>> raise ValueError('multi\n line\ndetail') +Traceback (most recent call last): + File "<stdin>", line 1, in ? +ValueError: multi + line +detail +\end{verbatim} + +The last three lines (starting with \exception{ValueError}) are +compared against the exception's type and detail, and the rest are +ignored. + +Best practice is to omit the traceback stack, unless it adds +significant documentation value to the example. So the last example +is probably better as: + +\begin{verbatim} +>>> raise ValueError('multi\n line\ndetail') +Traceback (most recent call last): + ... +ValueError: multi + line +detail +\end{verbatim} + +Note that tracebacks are treated very specially. In particular, in the +rewritten example, the use of \samp{...} is independent of doctest's +\constant{ELLIPSIS} option. The ellipsis in that example could be left +out, or could just as well be three (or three hundred) commas or digits, +or an indented transcript of a Monty Python skit. + +Some details you should read once, but won't need to remember: + +\begin{itemize} + +\item Doctest can't guess whether your expected output came from an + exception traceback or from ordinary printing. So, e.g., an example + that expects \samp{ValueError: 42 is prime} will pass whether + \exception{ValueError} is actually raised or if the example merely + prints that traceback text. In practice, ordinary output rarely begins + with a traceback header line, so this doesn't create real problems. + +\item Each line of the traceback stack (if present) must be indented + further than the first line of the example, \emph{or} start with a + non-alphanumeric character. The first line following the traceback + header indented the same and starting with an alphanumeric is taken + to be the start of the exception detail. Of course this does the + right thing for genuine tracebacks. + +\item When the \constant{IGNORE_EXCEPTION_DETAIL} doctest option is + is specified, everything following the leftmost colon is ignored. + +\item The interactive shell omits the traceback header line for some + \exception{SyntaxError}s. But doctest uses the traceback header + line to distinguish exceptions from non-exceptions. So in the rare + case where you need to test a \exception{SyntaxError} that omits the + traceback header, you will need to manually add the traceback header + line to your test example. + +\item For some \exception{SyntaxError}s, Python displays the character + position of the syntax error, using a \code{\^} marker: + +\begin{verbatim} +>>> 1 1 + File "<stdin>", line 1 + 1 1 + ^ +SyntaxError: invalid syntax +\end{verbatim} + + Since the lines showing the position of the error come before the + exception type and detail, they are not checked by doctest. For + example, the following test would pass, even though it puts the + \code{\^} marker in the wrong location: + +\begin{verbatim} +>>> 1 1 +Traceback (most recent call last): + File "<stdin>", line 1 + 1 1 + ^ +SyntaxError: invalid syntax +\end{verbatim} + +\end{itemize} + +\versionchanged[The ability to handle a multi-line exception detail, + and the \constant{IGNORE_EXCEPTION_DETAIL} doctest option, + were added]{2.4} + +\subsubsection{Option Flags and Directives\label{doctest-options}} + +A number of option flags control various aspects of doctest's +behavior. Symbolic names for the flags are supplied as module constants, +which can be or'ed together and passed to various functions. The names +can also be used in doctest directives (see below). + +The first group of options define test semantics, controlling +aspects of how doctest decides whether actual output matches an +example's expected output: + +\begin{datadesc}{DONT_ACCEPT_TRUE_FOR_1} + By default, if an expected output block contains just \code{1}, + an actual output block containing just \code{1} or just + \code{True} is considered to be a match, and similarly for \code{0} + versus \code{False}. When \constant{DONT_ACCEPT_TRUE_FOR_1} is + specified, neither substitution is allowed. The default behavior + caters to that Python changed the return type of many functions + from integer to boolean; doctests expecting "little integer" + output still work in these cases. This option will probably go + away, but not for several years. +\end{datadesc} + +\begin{datadesc}{DONT_ACCEPT_BLANKLINE} + By default, if an expected output block contains a line + containing only the string \code{<BLANKLINE>}, then that line + will match a blank line in the actual output. Because a + genuinely blank line delimits the expected output, this is + the only way to communicate that a blank line is expected. When + \constant{DONT_ACCEPT_BLANKLINE} is specified, this substitution + is not allowed. +\end{datadesc} + +\begin{datadesc}{NORMALIZE_WHITESPACE} + When specified, all sequences of whitespace (blanks and newlines) are + treated as equal. Any sequence of whitespace within the expected + output will match any sequence of whitespace within the actual output. + By default, whitespace must match exactly. + \constant{NORMALIZE_WHITESPACE} is especially useful when a line + of expected output is very long, and you want to wrap it across + multiple lines in your source. +\end{datadesc} + +\begin{datadesc}{ELLIPSIS} + When specified, an ellipsis marker (\code{...}) in the expected output + can match any substring in the actual output. This includes + substrings that span line boundaries, and empty substrings, so it's + best to keep usage of this simple. Complicated uses can lead to the + same kinds of "oops, it matched too much!" surprises that \regexp{.*} + is prone to in regular expressions. +\end{datadesc} + +\begin{datadesc}{IGNORE_EXCEPTION_DETAIL} + When specified, an example that expects an exception passes if + an exception of the expected type is raised, even if the exception + detail does not match. For example, an example expecting + \samp{ValueError: 42} will pass if the actual exception raised is + \samp{ValueError: 3*14}, but will fail, e.g., if + \exception{TypeError} is raised. + + Note that a similar effect can be obtained using \constant{ELLIPSIS}, + and \constant{IGNORE_EXCEPTION_DETAIL} may go away when Python releases + prior to 2.4 become uninteresting. Until then, + \constant{IGNORE_EXCEPTION_DETAIL} is the only clear way to write a + doctest that doesn't care about the exception detail yet continues + to pass under Python releases prior to 2.4 (doctest directives + appear to be comments to them). For example, + +\begin{verbatim} +>>> (1, 2)[3] = 'moo' #doctest: +IGNORE_EXCEPTION_DETAIL +Traceback (most recent call last): + File "<stdin>", line 1, in ? +TypeError: object doesn't support item assignment +\end{verbatim} + + passes under Python 2.4 and Python 2.3. The detail changed in 2.4, + to say "does not" instead of "doesn't". + +\end{datadesc} + +\begin{datadesc}{SKIP} + + When specified, do not run the example at all. This can be useful + in contexts where doctest examples serve as both documentation and + test cases, and an example should be included for documentation + purposes, but should not be checked. E.g., the example's output + might be random; or the example might depend on resources which + would be unavailable to the test driver. + + The SKIP flag can also be used for temporarily "commenting out" + examples. + +\end{datadesc} + +\begin{datadesc}{COMPARISON_FLAGS} + A bitmask or'ing together all the comparison flags above. +\end{datadesc} + +The second group of options controls how test failures are reported: + +\begin{datadesc}{REPORT_UDIFF} + When specified, failures that involve multi-line expected and + actual outputs are displayed using a unified diff. +\end{datadesc} + +\begin{datadesc}{REPORT_CDIFF} + When specified, failures that involve multi-line expected and + actual outputs will be displayed using a context diff. +\end{datadesc} + +\begin{datadesc}{REPORT_NDIFF} + When specified, differences are computed by \code{difflib.Differ}, + using the same algorithm as the popular \file{ndiff.py} utility. + This is the only method that marks differences within lines as + well as across lines. For example, if a line of expected output + contains digit \code{1} where actual output contains letter \code{l}, + a line is inserted with a caret marking the mismatching column + positions. +\end{datadesc} + +\begin{datadesc}{REPORT_ONLY_FIRST_FAILURE} + When specified, display the first failing example in each doctest, + but suppress output for all remaining examples. This will prevent + doctest from reporting correct examples that break because of + earlier failures; but it might also hide incorrect examples that + fail independently of the first failure. When + \constant{REPORT_ONLY_FIRST_FAILURE} is specified, the remaining + examples are still run, and still count towards the total number of + failures reported; only the output is suppressed. +\end{datadesc} + +\begin{datadesc}{REPORTING_FLAGS} + A bitmask or'ing together all the reporting flags above. +\end{datadesc} + +"Doctest directives" may be used to modify the option flags for +individual examples. Doctest directives are expressed as a special +Python comment following an example's source code: + +\begin{productionlist}[doctest] + \production{directive} + {"\#" "doctest:" \token{directive_options}} + \production{directive_options} + {\token{directive_option} ("," \token{directive_option})*} + \production{directive_option} + {\token{on_or_off} \token{directive_option_name}} + \production{on_or_off} + {"+" | "-"} + \production{directive_option_name} + {"DONT_ACCEPT_BLANKLINE" | "NORMALIZE_WHITESPACE" | ...} +\end{productionlist} + +Whitespace is not allowed between the \code{+} or \code{-} and the +directive option name. The directive option name can be any of the +option flag names explained above. + +An example's doctest directives modify doctest's behavior for that +single example. Use \code{+} to enable the named behavior, or +\code{-} to disable it. + +For example, this test passes: + +\begin{verbatim} +>>> print range(20) #doctest: +NORMALIZE_WHITESPACE +[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, +10, 11, 12, 13, 14, 15, 16, 17, 18, 19] +\end{verbatim} + +Without the directive it would fail, both because the actual output +doesn't have two blanks before the single-digit list elements, and +because the actual output is on a single line. This test also passes, +and also requires a directive to do so: + +\begin{verbatim} +>>> print range(20) # doctest:+ELLIPSIS +[0, 1, ..., 18, 19] +\end{verbatim} + +Multiple directives can be used on a single physical line, separated +by commas: + +\begin{verbatim} +>>> print range(20) # doctest: +ELLIPSIS, +NORMALIZE_WHITESPACE +[0, 1, ..., 18, 19] +\end{verbatim} + +If multiple directive comments are used for a single example, then +they are combined: + +\begin{verbatim} +>>> print range(20) # doctest: +ELLIPSIS +... # doctest: +NORMALIZE_WHITESPACE +[0, 1, ..., 18, 19] +\end{verbatim} + +As the previous example shows, you can add \samp{...} lines to your +example containing only directives. This can be useful when an +example is too long for a directive to comfortably fit on the same +line: + +\begin{verbatim} +>>> print range(5) + range(10,20) + range(30,40) + range(50,60) +... # doctest: +ELLIPSIS +[0, ..., 4, 10, ..., 19, 30, ..., 39, 50, ..., 59] +\end{verbatim} + +Note that since all options are disabled by default, and directives apply +only to the example they appear in, enabling options (via \code{+} in a +directive) is usually the only meaningful choice. However, option flags +can also be passed to functions that run doctests, establishing different +defaults. In such cases, disabling an option via \code{-} in a directive +can be useful. + +\versionchanged[Constants \constant{DONT_ACCEPT_BLANKLINE}, + \constant{NORMALIZE_WHITESPACE}, \constant{ELLIPSIS}, + \constant{IGNORE_EXCEPTION_DETAIL}, + \constant{REPORT_UDIFF}, \constant{REPORT_CDIFF}, + \constant{REPORT_NDIFF}, \constant{REPORT_ONLY_FIRST_FAILURE}, + \constant{COMPARISON_FLAGS} and \constant{REPORTING_FLAGS} + were added; by default \code{<BLANKLINE>} in expected output + matches an empty line in actual output; and doctest directives + were added]{2.4} +\versionchanged[Constant \constant{SKIP} was added]{2.5} + +There's also a way to register new option flag names, although this +isn't useful unless you intend to extend \refmodule{doctest} internals +via subclassing: + +\begin{funcdesc}{register_optionflag}{name} + Create a new option flag with a given name, and return the new + flag's integer value. \function{register_optionflag()} can be + used when subclassing \class{OutputChecker} or + \class{DocTestRunner} to create new options that are supported by + your subclasses. \function{register_optionflag} should always be + called using the following idiom: + +\begin{verbatim} + MY_FLAG = register_optionflag('MY_FLAG') +\end{verbatim} + + \versionadded{2.4} +\end{funcdesc} + +\subsubsection{Warnings\label{doctest-warnings}} + +\refmodule{doctest} is serious about requiring exact matches in expected +output. If even a single character doesn't match, the test fails. This +will probably surprise you a few times, as you learn exactly what Python +does and doesn't guarantee about output. For example, when printing a +dict, Python doesn't guarantee that the key-value pairs will be printed +in any particular order, so a test like + +% Hey! What happened to Monty Python examples? +% Tim: ask Guido -- it's his example! +\begin{verbatim} +>>> foo() +{"Hermione": "hippogryph", "Harry": "broomstick"} +\end{verbatim} + +is vulnerable! One workaround is to do + +\begin{verbatim} +>>> foo() == {"Hermione": "hippogryph", "Harry": "broomstick"} +True +\end{verbatim} + +instead. Another is to do + +\begin{verbatim} +>>> d = foo().items() +>>> d.sort() +>>> d +[('Harry', 'broomstick'), ('Hermione', 'hippogryph')] +\end{verbatim} + +There are others, but you get the idea. + +Another bad idea is to print things that embed an object address, like + +\begin{verbatim} +>>> id(1.0) # certain to fail some of the time +7948648 +>>> class C: pass +>>> C() # the default repr() for instances embeds an address +<__main__.C instance at 0x00AC18F0> +\end{verbatim} + +The \constant{ELLIPSIS} directive gives a nice approach for the last +example: + +\begin{verbatim} +>>> C() #doctest: +ELLIPSIS +<__main__.C instance at 0x...> +\end{verbatim} + +Floating-point numbers are also subject to small output variations across +platforms, because Python defers to the platform C library for float +formatting, and C libraries vary widely in quality here. + +\begin{verbatim} +>>> 1./7 # risky +0.14285714285714285 +>>> print 1./7 # safer +0.142857142857 +>>> print round(1./7, 6) # much safer +0.142857 +\end{verbatim} + +Numbers of the form \code{I/2.**J} are safe across all platforms, and I +often contrive doctest examples to produce numbers of that form: + +\begin{verbatim} +>>> 3./4 # utterly safe +0.75 +\end{verbatim} + +Simple fractions are also easier for people to understand, and that makes +for better documentation. + +\subsection{Basic API\label{doctest-basic-api}} + +The functions \function{testmod()} and \function{testfile()} provide a +simple interface to doctest that should be sufficient for most basic +uses. For a less formal introduction to these two functions, see +sections \ref{doctest-simple-testmod} and +\ref{doctest-simple-testfile}. + +\begin{funcdesc}{testfile}{filename\optional{, module_relative}\optional{, + name}\optional{, package}\optional{, + globs}\optional{, verbose}\optional{, + report}\optional{, optionflags}\optional{, + extraglobs}\optional{, raise_on_error}\optional{, + parser}\optional{, encoding}} + + All arguments except \var{filename} are optional, and should be + specified in keyword form. + + Test examples in the file named \var{filename}. Return + \samp{(\var{failure_count}, \var{test_count})}. + + Optional argument \var{module_relative} specifies how the filename + should be interpreted: + + \begin{itemize} + \item If \var{module_relative} is \code{True} (the default), then + \var{filename} specifies an OS-independent module-relative + path. By default, this path is relative to the calling + module's directory; but if the \var{package} argument is + specified, then it is relative to that package. To ensure + OS-independence, \var{filename} should use \code{/} characters + to separate path segments, and may not be an absolute path + (i.e., it may not begin with \code{/}). + \item If \var{module_relative} is \code{False}, then \var{filename} + specifies an OS-specific path. The path may be absolute or + relative; relative paths are resolved with respect to the + current working directory. + \end{itemize} + + Optional argument \var{name} gives the name of the test; by default, + or if \code{None}, \code{os.path.basename(\var{filename})} is used. + + Optional argument \var{package} is a Python package or the name of a + Python package whose directory should be used as the base directory + for a module-relative filename. If no package is specified, then + the calling module's directory is used as the base directory for + module-relative filenames. It is an error to specify \var{package} + if \var{module_relative} is \code{False}. + + Optional argument \var{globs} gives a dict to be used as the globals + when executing examples. A new shallow copy of this dict is + created for the doctest, so its examples start with a clean slate. + By default, or if \code{None}, a new empty dict is used. + + Optional argument \var{extraglobs} gives a dict merged into the + globals used to execute examples. This works like + \method{dict.update()}: if \var{globs} and \var{extraglobs} have a + common key, the associated value in \var{extraglobs} appears in the + combined dict. By default, or if \code{None}, no extra globals are + used. This is an advanced feature that allows parameterization of + doctests. For example, a doctest can be written for a base class, using + a generic name for the class, then reused to test any number of + subclasses by passing an \var{extraglobs} dict mapping the generic + name to the subclass to be tested. + + Optional argument \var{verbose} prints lots of stuff if true, and prints + only failures if false; by default, or if \code{None}, it's true + if and only if \code{'-v'} is in \code{sys.argv}. + + Optional argument \var{report} prints a summary at the end when true, + else prints nothing at the end. In verbose mode, the summary is + detailed, else the summary is very brief (in fact, empty if all tests + passed). + + Optional argument \var{optionflags} or's together option flags. See + section~\ref{doctest-options}. + + Optional argument \var{raise_on_error} defaults to false. If true, + an exception is raised upon the first failure or unexpected exception + in an example. This allows failures to be post-mortem debugged. + Default behavior is to continue running examples. + + Optional argument \var{parser} specifies a \class{DocTestParser} (or + subclass) that should be used to extract tests from the files. It + defaults to a normal parser (i.e., \code{\class{DocTestParser}()}). + + Optional argument \var{encoding} specifies an encoding that should + be used to convert the file to unicode. + + \versionadded{2.4} + + \versionchanged[The parameter \var{encoding} was added]{2.5} + +\end{funcdesc} + +\begin{funcdesc}{testmod}{\optional{m}\optional{, name}\optional{, + globs}\optional{, verbose}\optional{, + report}\optional{, + optionflags}\optional{, extraglobs}\optional{, + raise_on_error}\optional{, exclude_empty}} + + All arguments are optional, and all except for \var{m} should be + specified in keyword form. + + Test examples in docstrings in functions and classes reachable + from module \var{m} (or module \module{__main__} if \var{m} is not + supplied or is \code{None}), starting with \code{\var{m}.__doc__}. + + Also test examples reachable from dict \code{\var{m}.__test__}, if it + exists and is not \code{None}. \code{\var{m}.__test__} maps + names (strings) to functions, classes and strings; function and class + docstrings are searched for examples; strings are searched directly, + as if they were docstrings. + + Only docstrings attached to objects belonging to module \var{m} are + searched. + + Return \samp{(\var{failure_count}, \var{test_count})}. + + Optional argument \var{name} gives the name of the module; by default, + or if \code{None}, \code{\var{m}.__name__} is used. + + Optional argument \var{exclude_empty} defaults to false. If true, + objects for which no doctests are found are excluded from consideration. + The default is a backward compatibility hack, so that code still + using \method{doctest.master.summarize()} in conjunction with + \function{testmod()} continues to get output for objects with no tests. + The \var{exclude_empty} argument to the newer \class{DocTestFinder} + constructor defaults to true. + + Optional arguments \var{extraglobs}, \var{verbose}, \var{report}, + \var{optionflags}, \var{raise_on_error}, and \var{globs} are the same as + for function \function{testfile()} above, except that \var{globs} + defaults to \code{\var{m}.__dict__}. + + \versionchanged[The parameter \var{optionflags} was added]{2.3} + + \versionchanged[The parameters \var{extraglobs}, \var{raise_on_error} + and \var{exclude_empty} were added]{2.4} + + \versionchanged[The optional argument \var{isprivate}, deprecated + in 2.4, was removed]{2.5} + +\end{funcdesc} + +There's also a function to run the doctests associated with a single object. +This function is provided for backward compatibility. There are no plans +to deprecate it, but it's rarely useful: + +\begin{funcdesc}{run_docstring_examples}{f, globs\optional{, + verbose}\optional{, name}\optional{, + compileflags}\optional{, optionflags}} + + Test examples associated with object \var{f}; for example, \var{f} may + be a module, function, or class object. + + A shallow copy of dictionary argument \var{globs} is used for the + execution context. + + Optional argument \var{name} is used in failure messages, and defaults + to \code{"NoName"}. + + If optional argument \var{verbose} is true, output is generated even + if there are no failures. By default, output is generated only in case + of an example failure. + + Optional argument \var{compileflags} gives the set of flags that should + be used by the Python compiler when running the examples. By default, or + if \code{None}, flags are deduced corresponding to the set of future + features found in \var{globs}. + + Optional argument \var{optionflags} works as for function + \function{testfile()} above. +\end{funcdesc} + +\subsection{Unittest API\label{doctest-unittest-api}} + +As your collection of doctest'ed modules grows, you'll want a way to run +all their doctests systematically. Prior to Python 2.4, \refmodule{doctest} +had a barely documented \class{Tester} class that supplied a rudimentary +way to combine doctests from multiple modules. \class{Tester} was feeble, +and in practice most serious Python testing frameworks build on the +\refmodule{unittest} module, which supplies many flexible ways to combine +tests from multiple sources. So, in Python 2.4, \refmodule{doctest}'s +\class{Tester} class is deprecated, and \refmodule{doctest} provides two +functions that can be used to create \refmodule{unittest} test suites from +modules and text files containing doctests. These test suites can then be +run using \refmodule{unittest} test runners: + +\begin{verbatim} +import unittest +import doctest +import my_module_with_doctests, and_another + +suite = unittest.TestSuite() +for mod in my_module_with_doctests, and_another: + suite.addTest(doctest.DocTestSuite(mod)) +runner = unittest.TextTestRunner() +runner.run(suite) +\end{verbatim} + +There are two main functions for creating \class{\refmodule{unittest}.TestSuite} +instances from text files and modules with doctests: + +\begin{funcdesc}{DocFileSuite}{\optional{module_relative}\optional{, + package}\optional{, setUp}\optional{, + tearDown}\optional{, globs}\optional{, + optionflags}\optional{, parser}\optional{, + encoding}} + + Convert doctest tests from one or more text files to a + \class{\refmodule{unittest}.TestSuite}. + + The returned \class{\refmodule{unittest}.TestSuite} is to be run by the + unittest framework and runs the interactive examples in each file. If an + example in any file fails, then the synthesized unit test fails, and a + \exception{failureException} exception is raised showing the name of the + file containing the test and a (sometimes approximate) line number. + + Pass one or more paths (as strings) to text files to be examined. + + Options may be provided as keyword arguments: + + Optional argument \var{module_relative} specifies how + the filenames in \var{paths} should be interpreted: + + \begin{itemize} + \item If \var{module_relative} is \code{True} (the default), then + each filename specifies an OS-independent module-relative + path. By default, this path is relative to the calling + module's directory; but if the \var{package} argument is + specified, then it is relative to that package. To ensure + OS-independence, each filename should use \code{/} characters + to separate path segments, and may not be an absolute path + (i.e., it may not begin with \code{/}). + \item If \var{module_relative} is \code{False}, then each filename + specifies an OS-specific path. The path may be absolute or + relative; relative paths are resolved with respect to the + current working directory. + \end{itemize} + + Optional argument \var{package} is a Python package or the name + of a Python package whose directory should be used as the base + directory for module-relative filenames. If no package is + specified, then the calling module's directory is used as the base + directory for module-relative filenames. It is an error to specify + \var{package} if \var{module_relative} is \code{False}. + + Optional argument \var{setUp} specifies a set-up function for + the test suite. This is called before running the tests in each + file. The \var{setUp} function will be passed a \class{DocTest} + object. The setUp function can access the test globals as the + \var{globs} attribute of the test passed. + + Optional argument \var{tearDown} specifies a tear-down function + for the test suite. This is called after running the tests in each + file. The \var{tearDown} function will be passed a \class{DocTest} + object. The setUp function can access the test globals as the + \var{globs} attribute of the test passed. + + Optional argument \var{globs} is a dictionary containing the + initial global variables for the tests. A new copy of this + dictionary is created for each test. By default, \var{globs} is + a new empty dictionary. + + Optional argument \var{optionflags} specifies the default + doctest options for the tests, created by or-ing together + individual option flags. See section~\ref{doctest-options}. + See function \function{set_unittest_reportflags()} below for + a better way to set reporting options. + + Optional argument \var{parser} specifies a \class{DocTestParser} (or + subclass) that should be used to extract tests from the files. It + defaults to a normal parser (i.e., \code{\class{DocTestParser}()}). + + Optional argument \var{encoding} specifies an encoding that should + be used to convert the file to unicode. + + \versionadded{2.4} + + \versionchanged[The global \code{__file__} was added to the + globals provided to doctests loaded from a text file using + \function{DocFileSuite()}]{2.5} + + \versionchanged[The parameter \var{encoding} was added]{2.5} + +\end{funcdesc} + +\begin{funcdesc}{DocTestSuite}{\optional{module}\optional{, + globs}\optional{, extraglobs}\optional{, + test_finder}\optional{, setUp}\optional{, + tearDown}\optional{, checker}} + Convert doctest tests for a module to a + \class{\refmodule{unittest}.TestSuite}. + + The returned \class{\refmodule{unittest}.TestSuite} is to be run by the + unittest framework and runs each doctest in the module. If any of the + doctests fail, then the synthesized unit test fails, and a + \exception{failureException} exception is raised showing the name of the + file containing the test and a (sometimes approximate) line number. + + Optional argument \var{module} provides the module to be tested. It + can be a module object or a (possibly dotted) module name. If not + specified, the module calling this function is used. + + Optional argument \var{globs} is a dictionary containing the + initial global variables for the tests. A new copy of this + dictionary is created for each test. By default, \var{globs} is + a new empty dictionary. + + Optional argument \var{extraglobs} specifies an extra set of + global variables, which is merged into \var{globs}. By default, no + extra globals are used. + + Optional argument \var{test_finder} is the \class{DocTestFinder} + object (or a drop-in replacement) that is used to extract doctests + from the module. + + Optional arguments \var{setUp}, \var{tearDown}, and \var{optionflags} + are the same as for function \function{DocFileSuite()} above. + + \versionadded{2.3} + + \versionchanged[The parameters \var{globs}, \var{extraglobs}, + \var{test_finder}, \var{setUp}, \var{tearDown}, and + \var{optionflags} were added; this function now uses the same search + technique as \function{testmod()}]{2.4} +\end{funcdesc} + +Under the covers, \function{DocTestSuite()} creates a +\class{\refmodule{unittest}.TestSuite} out of \class{doctest.DocTestCase} +instances, and \class{DocTestCase} is a subclass of +\class{\refmodule{unittest}.TestCase}. \class{DocTestCase} isn't documented +here (it's an internal detail), but studying its code can answer questions +about the exact details of \refmodule{unittest} integration. + +Similarly, \function{DocFileSuite()} creates a +\class{\refmodule{unittest}.TestSuite} out of \class{doctest.DocFileCase} +instances, and \class{DocFileCase} is a subclass of \class{DocTestCase}. + +So both ways of creating a \class{\refmodule{unittest}.TestSuite} run +instances of \class{DocTestCase}. This is important for a subtle reason: +when you run \refmodule{doctest} functions yourself, you can control the +\refmodule{doctest} options in use directly, by passing option flags to +\refmodule{doctest} functions. However, if you're writing a +\refmodule{unittest} framework, \refmodule{unittest} ultimately controls +when and how tests get run. The framework author typically wants to +control \refmodule{doctest} reporting options (perhaps, e.g., specified by +command line options), but there's no way to pass options through +\refmodule{unittest} to \refmodule{doctest} test runners. + +For this reason, \refmodule{doctest} also supports a notion of +\refmodule{doctest} reporting flags specific to \refmodule{unittest} +support, via this function: + +\begin{funcdesc}{set_unittest_reportflags}{flags} + Set the \refmodule{doctest} reporting flags to use. + + Argument \var{flags} or's together option flags. See + section~\ref{doctest-options}. Only "reporting flags" can be used. + + This is a module-global setting, and affects all future doctests run by + module \refmodule{unittest}: the \method{runTest()} method of + \class{DocTestCase} looks at the option flags specified for the test case + when the \class{DocTestCase} instance was constructed. If no reporting + flags were specified (which is the typical and expected case), + \refmodule{doctest}'s \refmodule{unittest} reporting flags are or'ed into + the option flags, and the option flags so augmented are passed to the + \class{DocTestRunner} instance created to run the doctest. If any + reporting flags were specified when the \class{DocTestCase} instance was + constructed, \refmodule{doctest}'s \refmodule{unittest} reporting flags + are ignored. + + The value of the \refmodule{unittest} reporting flags in effect before the + function was called is returned by the function. + + \versionadded{2.4} +\end{funcdesc} + + +\subsection{Advanced API\label{doctest-advanced-api}} + +The basic API is a simple wrapper that's intended to make doctest easy +to use. It is fairly flexible, and should meet most users' needs; +however, if you require more fine-grained control over testing, or +wish to extend doctest's capabilities, then you should use the +advanced API. + +The advanced API revolves around two container classes, which are used +to store the interactive examples extracted from doctest cases: + +\begin{itemize} +\item \class{Example}: A single python statement, paired with its + expected output. +\item \class{DocTest}: A collection of \class{Example}s, typically + extracted from a single docstring or text file. +\end{itemize} + +Additional processing classes are defined to find, parse, and run, and +check doctest examples: + +\begin{itemize} +\item \class{DocTestFinder}: Finds all docstrings in a given module, + and uses a \class{DocTestParser} to create a \class{DocTest} + from every docstring that contains interactive examples. +\item \class{DocTestParser}: Creates a \class{DocTest} object from + a string (such as an object's docstring). +\item \class{DocTestRunner}: Executes the examples in a + \class{DocTest}, and uses an \class{OutputChecker} to verify + their output. +\item \class{OutputChecker}: Compares the actual output from a + doctest example with the expected output, and decides whether + they match. +\end{itemize} + +The relationships among these processing classes are summarized in the +following diagram: + +\begin{verbatim} + list of: ++------+ +---------+ +|module| --DocTestFinder-> | DocTest | --DocTestRunner-> results ++------+ | ^ +---------+ | ^ (printed) + | | | Example | | | + v | | ... | v | + DocTestParser | Example | OutputChecker + +---------+ +\end{verbatim} + +\subsubsection{DocTest Objects\label{doctest-DocTest}} +\begin{classdesc}{DocTest}{examples, globs, name, filename, lineno, + docstring} + A collection of doctest examples that should be run in a single + namespace. The constructor arguments are used to initialize the + member variables of the same names. + \versionadded{2.4} +\end{classdesc} + +\class{DocTest} defines the following member variables. They are +initialized by the constructor, and should not be modified directly. + +\begin{memberdesc}{examples} + A list of \class{Example} objects encoding the individual + interactive Python examples that should be run by this test. +\end{memberdesc} + +\begin{memberdesc}{globs} + The namespace (aka globals) that the examples should be run in. + This is a dictionary mapping names to values. Any changes to the + namespace made by the examples (such as binding new variables) + will be reflected in \member{globs} after the test is run. +\end{memberdesc} + +\begin{memberdesc}{name} + A string name identifying the \class{DocTest}. Typically, this is + the name of the object or file that the test was extracted from. +\end{memberdesc} + +\begin{memberdesc}{filename} + The name of the file that this \class{DocTest} was extracted from; + or \code{None} if the filename is unknown, or if the + \class{DocTest} was not extracted from a file. +\end{memberdesc} + +\begin{memberdesc}{lineno} + The line number within \member{filename} where this + \class{DocTest} begins, or \code{None} if the line number is + unavailable. This line number is zero-based with respect to the + beginning of the file. +\end{memberdesc} + +\begin{memberdesc}{docstring} + The string that the test was extracted from, or `None` if the + string is unavailable, or if the test was not extracted from a + string. +\end{memberdesc} + +\subsubsection{Example Objects\label{doctest-Example}} +\begin{classdesc}{Example}{source, want\optional{, + exc_msg}\optional{, lineno}\optional{, + indent}\optional{, options}} + A single interactive example, consisting of a Python statement and + its expected output. The constructor arguments are used to + initialize the member variables of the same names. + \versionadded{2.4} +\end{classdesc} + +\class{Example} defines the following member variables. They are +initialized by the constructor, and should not be modified directly. + +\begin{memberdesc}{source} + A string containing the example's source code. This source code + consists of a single Python statement, and always ends with a + newline; the constructor adds a newline when necessary. +\end{memberdesc} + +\begin{memberdesc}{want} + The expected output from running the example's source code (either + from stdout, or a traceback in case of exception). \member{want} + ends with a newline unless no output is expected, in which case + it's an empty string. The constructor adds a newline when + necessary. +\end{memberdesc} + +\begin{memberdesc}{exc_msg} + The exception message generated by the example, if the example is + expected to generate an exception; or \code{None} if it is not + expected to generate an exception. This exception message is + compared against the return value of + \function{traceback.format_exception_only()}. \member{exc_msg} + ends with a newline unless it's \code{None}. The constructor adds + a newline if needed. +\end{memberdesc} + +\begin{memberdesc}{lineno} + The line number within the string containing this example where + the example begins. This line number is zero-based with respect + to the beginning of the containing string. +\end{memberdesc} + +\begin{memberdesc}{indent} + The example's indentation in the containing string, i.e., the + number of space characters that precede the example's first + prompt. +\end{memberdesc} + +\begin{memberdesc}{options} + A dictionary mapping from option flags to \code{True} or + \code{False}, which is used to override default options for this + example. Any option flags not contained in this dictionary are + left at their default value (as specified by the + \class{DocTestRunner}'s \member{optionflags}). + By default, no options are set. +\end{memberdesc} + +\subsubsection{DocTestFinder objects\label{doctest-DocTestFinder}} +\begin{classdesc}{DocTestFinder}{\optional{verbose}\optional{, + parser}\optional{, recurse}\optional{, + exclude_empty}} + A processing class used to extract the \class{DocTest}s that are + relevant to a given object, from its docstring and the docstrings + of its contained objects. \class{DocTest}s can currently be + extracted from the following object types: modules, functions, + classes, methods, staticmethods, classmethods, and properties. + + The optional argument \var{verbose} can be used to display the + objects searched by the finder. It defaults to \code{False} (no + output). + + The optional argument \var{parser} specifies the + \class{DocTestParser} object (or a drop-in replacement) that is + used to extract doctests from docstrings. + + If the optional argument \var{recurse} is false, then + \method{DocTestFinder.find()} will only examine the given object, + and not any contained objects. + + If the optional argument \var{exclude_empty} is false, then + \method{DocTestFinder.find()} will include tests for objects with + empty docstrings. + + \versionadded{2.4} +\end{classdesc} + +\class{DocTestFinder} defines the following method: + +\begin{methoddesc}{find}{obj\optional{, name}\optional{, + module}\optional{, globs}\optional{, extraglobs}} + Return a list of the \class{DocTest}s that are defined by + \var{obj}'s docstring, or by any of its contained objects' + docstrings. + + The optional argument \var{name} specifies the object's name; this + name will be used to construct names for the returned + \class{DocTest}s. If \var{name} is not specified, then + \code{\var{obj}.__name__} is used. + + The optional parameter \var{module} is the module that contains + the given object. If the module is not specified or is None, then + the test finder will attempt to automatically determine the + correct module. The object's module is used: + + \begin{itemize} + \item As a default namespace, if \var{globs} is not specified. + \item To prevent the DocTestFinder from extracting DocTests + from objects that are imported from other modules. (Contained + objects with modules other than \var{module} are ignored.) + \item To find the name of the file containing the object. + \item To help find the line number of the object within its file. + \end{itemize} + + If \var{module} is \code{False}, no attempt to find the module + will be made. This is obscure, of use mostly in testing doctest + itself: if \var{module} is \code{False}, or is \code{None} but + cannot be found automatically, then all objects are considered to + belong to the (non-existent) module, so all contained objects will + (recursively) be searched for doctests. + + The globals for each \class{DocTest} is formed by combining + \var{globs} and \var{extraglobs} (bindings in \var{extraglobs} + override bindings in \var{globs}). A new shallow copy of the globals + dictionary is created for each \class{DocTest}. If \var{globs} is + not specified, then it defaults to the module's \var{__dict__}, if + specified, or \code{\{\}} otherwise. If \var{extraglobs} is not + specified, then it defaults to \code{\{\}}. +\end{methoddesc} + +\subsubsection{DocTestParser objects\label{doctest-DocTestParser}} +\begin{classdesc}{DocTestParser}{} + A processing class used to extract interactive examples from a + string, and use them to create a \class{DocTest} object. + \versionadded{2.4} +\end{classdesc} + +\class{DocTestParser} defines the following methods: + +\begin{methoddesc}{get_doctest}{string, globs, name, filename, lineno} + Extract all doctest examples from the given string, and collect + them into a \class{DocTest} object. + + \var{globs}, \var{name}, \var{filename}, and \var{lineno} are + attributes for the new \class{DocTest} object. See the + documentation for \class{DocTest} for more information. +\end{methoddesc} + +\begin{methoddesc}{get_examples}{string\optional{, name}} + Extract all doctest examples from the given string, and return + them as a list of \class{Example} objects. Line numbers are + 0-based. The optional argument \var{name} is a name identifying + this string, and is only used for error messages. +\end{methoddesc} + +\begin{methoddesc}{parse}{string\optional{, name}} + Divide the given string into examples and intervening text, and + return them as a list of alternating \class{Example}s and strings. + Line numbers for the \class{Example}s are 0-based. The optional + argument \var{name} is a name identifying this string, and is only + used for error messages. +\end{methoddesc} + +\subsubsection{DocTestRunner objects\label{doctest-DocTestRunner}} +\begin{classdesc}{DocTestRunner}{\optional{checker}\optional{, + verbose}\optional{, optionflags}} + A processing class used to execute and verify the interactive + examples in a \class{DocTest}. + + The comparison between expected outputs and actual outputs is done + by an \class{OutputChecker}. This comparison may be customized + with a number of option flags; see section~\ref{doctest-options} + for more information. If the option flags are insufficient, then + the comparison may also be customized by passing a subclass of + \class{OutputChecker} to the constructor. + + The test runner's display output can be controlled in two ways. + First, an output function can be passed to + \method{TestRunner.run()}; this function will be called with + strings that should be displayed. It defaults to + \code{sys.stdout.write}. If capturing the output is not + sufficient, then the display output can be also customized by + subclassing DocTestRunner, and overriding the methods + \method{report_start}, \method{report_success}, + \method{report_unexpected_exception}, and \method{report_failure}. + + The optional keyword argument \var{checker} specifies the + \class{OutputChecker} object (or drop-in replacement) that should + be used to compare the expected outputs to the actual outputs of + doctest examples. + + The optional keyword argument \var{verbose} controls the + \class{DocTestRunner}'s verbosity. If \var{verbose} is + \code{True}, then information is printed about each example, as it + is run. If \var{verbose} is \code{False}, then only failures are + printed. If \var{verbose} is unspecified, or \code{None}, then + verbose output is used iff the command-line switch \programopt{-v} + is used. + + The optional keyword argument \var{optionflags} can be used to + control how the test runner compares expected output to actual + output, and how it displays failures. For more information, see + section~\ref{doctest-options}. + + \versionadded{2.4} +\end{classdesc} + +\class{DocTestParser} defines the following methods: + +\begin{methoddesc}{report_start}{out, test, example} + Report that the test runner is about to process the given example. + This method is provided to allow subclasses of + \class{DocTestRunner} to customize their output; it should not be + called directly. + + \var{example} is the example about to be processed. \var{test} is + the test containing \var{example}. \var{out} is the output + function that was passed to \method{DocTestRunner.run()}. +\end{methoddesc} + +\begin{methoddesc}{report_success}{out, test, example, got} + Report that the given example ran successfully. This method is + provided to allow subclasses of \class{DocTestRunner} to customize + their output; it should not be called directly. + + \var{example} is the example about to be processed. \var{got} is + the actual output from the example. \var{test} is the test + containing \var{example}. \var{out} is the output function that + was passed to \method{DocTestRunner.run()}. +\end{methoddesc} + +\begin{methoddesc}{report_failure}{out, test, example, got} + Report that the given example failed. This method is provided to + allow subclasses of \class{DocTestRunner} to customize their + output; it should not be called directly. + + \var{example} is the example about to be processed. \var{got} is + the actual output from the example. \var{test} is the test + containing \var{example}. \var{out} is the output function that + was passed to \method{DocTestRunner.run()}. +\end{methoddesc} + +\begin{methoddesc}{report_unexpected_exception}{out, test, example, exc_info} + Report that the given example raised an unexpected exception. + This method is provided to allow subclasses of + \class{DocTestRunner} to customize their output; it should not be + called directly. + + \var{example} is the example about to be processed. + \var{exc_info} is a tuple containing information about the + unexpected exception (as returned by \function{sys.exc_info()}). + \var{test} is the test containing \var{example}. \var{out} is the + output function that was passed to \method{DocTestRunner.run()}. +\end{methoddesc} + +\begin{methoddesc}{run}{test\optional{, compileflags}\optional{, + out}\optional{, clear_globs}} + Run the examples in \var{test} (a \class{DocTest} object), and + display the results using the writer function \var{out}. + + The examples are run in the namespace \code{test.globs}. If + \var{clear_globs} is true (the default), then this namespace will + be cleared after the test runs, to help with garbage collection. + If you would like to examine the namespace after the test + completes, then use \var{clear_globs=False}. + + \var{compileflags} gives the set of flags that should be used by + the Python compiler when running the examples. If not specified, + then it will default to the set of future-import flags that apply + to \var{globs}. + + The output of each example is checked using the + \class{DocTestRunner}'s output checker, and the results are + formatted by the \method{DocTestRunner.report_*} methods. +\end{methoddesc} + +\begin{methoddesc}{summarize}{\optional{verbose}} + Print a summary of all the test cases that have been run by this + DocTestRunner, and return a tuple \samp{(\var{failure_count}, + \var{test_count})}. + + The optional \var{verbose} argument controls how detailed the + summary is. If the verbosity is not specified, then the + \class{DocTestRunner}'s verbosity is used. +\end{methoddesc} + +\subsubsection{OutputChecker objects\label{doctest-OutputChecker}} + +\begin{classdesc}{OutputChecker}{} + A class used to check the whether the actual output from a doctest + example matches the expected output. \class{OutputChecker} + defines two methods: \method{check_output}, which compares a given + pair of outputs, and returns true if they match; and + \method{output_difference}, which returns a string describing the + differences between two outputs. + \versionadded{2.4} +\end{classdesc} + +\class{OutputChecker} defines the following methods: + +\begin{methoddesc}{check_output}{want, got, optionflags} + Return \code{True} iff the actual output from an example + (\var{got}) matches the expected output (\var{want}). These + strings are always considered to match if they are identical; but + depending on what option flags the test runner is using, several + non-exact match types are also possible. See + section~\ref{doctest-options} for more information about option + flags. +\end{methoddesc} + +\begin{methoddesc}{output_difference}{example, got, optionflags} + Return a string describing the differences between the expected + output for a given example (\var{example}) and the actual output + (\var{got}). \var{optionflags} is the set of option flags used to + compare \var{want} and \var{got}. +\end{methoddesc} + +\subsection{Debugging\label{doctest-debugging}} + +Doctest provides several mechanisms for debugging doctest examples: + +\begin{itemize} +\item Several functions convert doctests to executable Python + programs, which can be run under the Python debugger, \refmodule{pdb}. +\item The \class{DebugRunner} class is a subclass of + \class{DocTestRunner} that raises an exception for the first + failing example, containing information about that example. + This information can be used to perform post-mortem debugging on + the example. +\item The \refmodule{unittest} cases generated by \function{DocTestSuite()} + support the \method{debug()} method defined by + \class{\refmodule{unittest}.TestCase}. +\item You can add a call to \function{\refmodule{pdb}.set_trace()} in a + doctest example, and you'll drop into the Python debugger when that + line is executed. Then you can inspect current values of variables, + and so on. For example, suppose \file{a.py} contains just this + module docstring: + +\begin{verbatim} +""" +>>> def f(x): +... g(x*2) +>>> def g(x): +... print x+3 +... import pdb; pdb.set_trace() +>>> f(3) +9 +""" +\end{verbatim} + + Then an interactive Python session may look like this: + +\begin{verbatim} +>>> import a, doctest +>>> doctest.testmod(a) +--Return-- +> <doctest a[1]>(3)g()->None +-> import pdb; pdb.set_trace() +(Pdb) list + 1 def g(x): + 2 print x+3 + 3 -> import pdb; pdb.set_trace() +[EOF] +(Pdb) print x +6 +(Pdb) step +--Return-- +> <doctest a[0]>(2)f()->None +-> g(x*2) +(Pdb) list + 1 def f(x): + 2 -> g(x*2) +[EOF] +(Pdb) print x +3 +(Pdb) step +--Return-- +> <doctest a[2]>(1)?()->None +-> f(3) +(Pdb) cont +(0, 3) +>>> +\end{verbatim} + + \versionchanged[The ability to use \code{\refmodule{pdb}.set_trace()} + usefully inside doctests was added]{2.4} +\end{itemize} + +Functions that convert doctests to Python code, and possibly run +the synthesized code under the debugger: + +\begin{funcdesc}{script_from_examples}{s} + Convert text with examples to a script. + + Argument \var{s} is a string containing doctest examples. The string + is converted to a Python script, where doctest examples in \var{s} + are converted to regular code, and everything else is converted to + Python comments. The generated script is returned as a string. + For example, + + \begin{verbatim} + import doctest + print doctest.script_from_examples(r""" + Set x and y to 1 and 2. + >>> x, y = 1, 2 + + Print their sum: + >>> print x+y + 3 + """) + \end{verbatim} + + displays: + + \begin{verbatim} + # Set x and y to 1 and 2. + x, y = 1, 2 + # + # Print their sum: + print x+y + # Expected: + ## 3 + \end{verbatim} + + This function is used internally by other functions (see below), but + can also be useful when you want to transform an interactive Python + session into a Python script. + + \versionadded{2.4} +\end{funcdesc} + +\begin{funcdesc}{testsource}{module, name} + Convert the doctest for an object to a script. + + Argument \var{module} is a module object, or dotted name of a module, + containing the object whose doctests are of interest. Argument + \var{name} is the name (within the module) of the object with the + doctests of interest. The result is a string, containing the + object's docstring converted to a Python script, as described for + \function{script_from_examples()} above. For example, if module + \file{a.py} contains a top-level function \function{f()}, then + +\begin{verbatim} +import a, doctest +print doctest.testsource(a, "a.f") +\end{verbatim} + + prints a script version of function \function{f()}'s docstring, + with doctests converted to code, and the rest placed in comments. + + \versionadded{2.3} +\end{funcdesc} + +\begin{funcdesc}{debug}{module, name\optional{, pm}} + Debug the doctests for an object. + + The \var{module} and \var{name} arguments are the same as for function + \function{testsource()} above. The synthesized Python script for the + named object's docstring is written to a temporary file, and then that + file is run under the control of the Python debugger, \refmodule{pdb}. + + A shallow copy of \code{\var{module}.__dict__} is used for both local + and global execution context. + + Optional argument \var{pm} controls whether post-mortem debugging is + used. If \var{pm} has a true value, the script file is run directly, and + the debugger gets involved only if the script terminates via raising an + unhandled exception. If it does, then post-mortem debugging is invoked, + via \code{\refmodule{pdb}.post_mortem()}, passing the traceback object + from the unhandled exception. If \var{pm} is not specified, or is false, + the script is run under the debugger from the start, via passing an + appropriate \function{execfile()} call to \code{\refmodule{pdb}.run()}. + + \versionadded{2.3} + + \versionchanged[The \var{pm} argument was added]{2.4} +\end{funcdesc} + +\begin{funcdesc}{debug_src}{src\optional{, pm}\optional{, globs}} + Debug the doctests in a string. + + This is like function \function{debug()} above, except that + a string containing doctest examples is specified directly, via + the \var{src} argument. + + Optional argument \var{pm} has the same meaning as in function + \function{debug()} above. + + Optional argument \var{globs} gives a dictionary to use as both + local and global execution context. If not specified, or \code{None}, + an empty dictionary is used. If specified, a shallow copy of the + dictionary is used. + + \versionadded{2.4} +\end{funcdesc} + +The \class{DebugRunner} class, and the special exceptions it may raise, +are of most interest to testing framework authors, and will only be +sketched here. See the source code, and especially \class{DebugRunner}'s +docstring (which is a doctest!) for more details: + +\begin{classdesc}{DebugRunner}{\optional{checker}\optional{, + verbose}\optional{, optionflags}} + + A subclass of \class{DocTestRunner} that raises an exception as + soon as a failure is encountered. If an unexpected exception + occurs, an \exception{UnexpectedException} exception is raised, + containing the test, the example, and the original exception. If + the output doesn't match, then a \exception{DocTestFailure} + exception is raised, containing the test, the example, and the + actual output. + + For information about the constructor parameters and methods, see + the documentation for \class{DocTestRunner} in + section~\ref{doctest-advanced-api}. +\end{classdesc} + +There are two exceptions that may be raised by \class{DebugRunner} +instances: + +\begin{excclassdesc}{DocTestFailure}{test, example, got} + An exception thrown by \class{DocTestRunner} to signal that a + doctest example's actual output did not match its expected output. + The constructor arguments are used to initialize the member + variables of the same names. +\end{excclassdesc} +\exception{DocTestFailure} defines the following member variables: +\begin{memberdesc}{test} + The \class{DocTest} object that was being run when the example failed. +\end{memberdesc} +\begin{memberdesc}{example} + The \class{Example} that failed. +\end{memberdesc} +\begin{memberdesc}{got} + The example's actual output. +\end{memberdesc} + +\begin{excclassdesc}{UnexpectedException}{test, example, exc_info} + An exception thrown by \class{DocTestRunner} to signal that a + doctest example raised an unexpected exception. The constructor + arguments are used to initialize the member variables of the same + names. +\end{excclassdesc} +\exception{UnexpectedException} defines the following member variables: +\begin{memberdesc}{test} + The \class{DocTest} object that was being run when the example failed. +\end{memberdesc} +\begin{memberdesc}{example} + The \class{Example} that failed. +\end{memberdesc} +\begin{memberdesc}{exc_info} + A tuple containing information about the unexpected exception, as + returned by \function{sys.exc_info()}. +\end{memberdesc} + +\subsection{Soapbox\label{doctest-soapbox}} + +As mentioned in the introduction, \refmodule{doctest} has grown to have +three primary uses: + +\begin{enumerate} +\item Checking examples in docstrings. +\item Regression testing. +\item Executable documentation / literate testing. +\end{enumerate} + +These uses have different requirements, and it is important to +distinguish them. In particular, filling your docstrings with obscure +test cases makes for bad documentation. + +When writing a docstring, choose docstring examples with care. +There's an art to this that needs to be learned---it may not be +natural at first. Examples should add genuine value to the +documentation. A good example can often be worth many words. +If done with care, the examples will be invaluable for your users, and +will pay back the time it takes to collect them many times over as the +years go by and things change. I'm still amazed at how often one of +my \refmodule{doctest} examples stops working after a "harmless" +change. + +Doctest also makes an excellent tool for regression testing, especially if +you don't skimp on explanatory text. By interleaving prose and examples, +it becomes much easier to keep track of what's actually being tested, and +why. When a test fails, good prose can make it much easier to figure out +what the problem is, and how it should be fixed. It's true that you could +write extensive comments in code-based testing, but few programmers do. +Many have found that using doctest approaches instead leads to much clearer +tests. Perhaps this is simply because doctest makes writing prose a little +easier than writing code, while writing comments in code is a little +harder. I think it goes deeper than just that: the natural attitude +when writing a doctest-based test is that you want to explain the fine +points of your software, and illustrate them with examples. This in +turn naturally leads to test files that start with the simplest features, +and logically progress to complications and edge cases. A coherent +narrative is the result, instead of a collection of isolated functions +that test isolated bits of functionality seemingly at random. It's +a different attitude, and produces different results, blurring the +distinction between testing and explaining. + +Regression testing is best confined to dedicated objects or files. There +are several options for organizing tests: + +\begin{itemize} +\item Write text files containing test cases as interactive examples, + and test the files using \function{testfile()} or + \function{DocFileSuite()}. This is recommended, although is + easiest to do for new projects, designed from the start to use + doctest. +\item Define functions named \code{_regrtest_\textit{topic}} that + consist of single docstrings, containing test cases for the + named topics. These functions can be included in the same file + as the module, or separated out into a separate test file. +\item Define a \code{__test__} dictionary mapping from regression test + topics to docstrings containing test cases. +\end{itemize} |